Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

Cordum Backend Capabilities (Redis + NATS)

This document tracks the current backend features, their status, and where they are exercised. It is code-accurate as of this commit.

Runtime Stack (active)​

  • Language: Go
  • Bus: NATS with optional JetStream durability (NATS_USE_JETSTREAM=1), DLQ emitter
  • State: Redis for job state, contexts/results, workflows/runs, config, DLQ
  • Artifacts: Redis-backed store with retention classes
  • Schemas: JSON schema registry + validation
  • Locks: Redis-backed shared/exclusive locks
  • Secrets: secret:// reference detection and redaction helpers (policy enforcement via risk tags/labels)

Components & Features​

Scheduler​

  • Dispatch: consumes sys.job.submit, routes to worker topics or direct subjects (least-loaded strategy).
  • States: Redis job store (atomic via WATCH), deadlines, trace linkage.
  • Safety: Safety client with half-open circuit breaker; decisions persisted (includes effective_config payload).
  • Registry: in-memory with TTL expiry loop to drop dead workers.
  • Reliability: JetStream mode uses explicit ack/nak on durable subjects; scheduler is idempotent under redelivery (per-job lock + retryable errors).
  • Reconciler: timeout scans for dispatched/running; bounded retries + lock-based to avoid double processing.
  • Pending replayer: replays PENDING jobs past dispatch timeout to avoid stuck jobs.
  • DLQ: emits to sys.job.dlq on failures.
  • Hints & cancel: respects preferred worker/pool hints via labels; broadcasts job cancel packets to sys.job.cancel (best-effort).

API Gateway​

  • Jobs: submit/list/get/cancel, trace fetch; list supports filters (state/topic/tenant/team/time/trace) and cursor pagination (cursor/next_cursor).
  • Workflows: create/upsert (/api/v1/workflows), list/get, runs start/get/list, cancel run, rerun, timeline.
  • Approvals: job approvals (/api/v1/approvals/...), including workflow gate approvals.
  • Config: Redis-backed config service (set/get via /api/v1/config, effective via /api/v1/config/effective).
  • Policy: evaluate/simulate/explain + snapshots, bundle list/detail/update, bundle snapshots, publish/rollback, audit (admin role enforced when enterprise RBAC is enabled).
  • Schemas: register/get/list/delete JSON schemas.
  • Locks: acquire/release/renew/get shared/exclusive locks.
  • Artifacts: put/get with retention class metadata.
  • Packs: install/list/show/verify/uninstall; pack registry stored in config service.
  • Marketplace: list catalogs + install by catalog/URL (cfg:system:pack_catalogs, seeded with the official catalog by default).
  • Stream: WS stream of bus packets (includes heartbeats and job events).
  • Memory: pointer reader (GET /api/v1/memory?ptr=...) used by admins or UI clients to inspect redis://ctx:*, redis://res:*, and redis://mem:* keys.
  • Security: CORS/WS origin allowlist (set CORDUM_ALLOWED_ORIGINS for non-local browser clients).
  • DLQ: list/delete/retry; retry rehydrates original context into a new job id and re-dispatches.

Workflow Engine Service​

  • Control plane: cmd/cordum-workflow-engine (binary cordum-workflow-engine) subscribes to sys.job.result (queue group) and advances runs independently from the gateway.
  • Storage: Redis workflows and runs (core/workflow), status indexes for reconciliation.
  • Execution: starts runs, dispatches ready steps as jobs (job ID = runID:stepID@attempt), consumes job results to advance run state.
  • Fan-out: for_each expression evaluated against run input/context; child jobs dispatched with index/item metadata; parent aggregated.
  • DAG deps: depends_on allows parallel independent steps; a step runs only after all dependencies succeed.
  • Failure semantics: failed/cancelled/timed-out deps block downstream steps (no implicit continue-on-error).
  • Dataflow: step input supports ${...} expressions; step outputs are recorded in run context under steps.<step_id> and optionally output_path.
  • Step types: built-ins (approval, delay, notify, condition) plus job-dispatch steps (any non-built-in type dispatches as a job via step.topic). Approval steps dispatch gate jobs to job.cordum.approval-gate (capsdk.SubjectWorkflowApprovalGate) that enter the unified job approval queue (the job-level approval gate uses the distinct sys.approval.gate topic).
  • Reliability: per-step retry/backoff (exponential), budget deadline hint from timeout_sec; job-result handling returns retryable errors (NAK) under transient store/lock failures so results aren’t dropped; reconciler replays terminal job states from JobStore and resumes delayed retries; tests cover fan-out/retry/approval/max_parallel.
  • Rerun: rerun-from-step and dry-run support.
  • Validation: workflow input schema validation; step input/output schema validation.
  • Audit: append-only timeline for run events.
  • Hooks: callbacks on step dispatch/finish for observability.
  • Routing: route labels/worker_id propagated to job labels for scheduler hints; cancel guard prevents further dispatch after cancel.

Config Service​

  • Redis-backed hierarchical merge (system→org→team→workflow→step) with shallow overrides.
  • Effective snapshot includes version and hash.
  • REST endpoints exposed via gateway.

DLQ​

  • Redis-backed DLQ store with add/list/delete/get; gateway exposes list/delete/retry and re-dispatches under a new job id.

Workers​

  • External workers (not in this repo) subscribe to sys.job.cancel and honor cancel requests.
  • CAP runtime SDK is exposed at sdk/runtime for typed handlers + pointer hydration; use CAP worker helpers for heartbeats/progress/cancel.

Pending/Next (to align with plan)​

  • DLQ ops: add pagination and richer telemetry.
  • Ops filters: server-side trace search and richer analytics/alerts.
  • Optional: vector store bindings, artifacts (S3), secrets (Vault/KMS) when step types require them.
  • CAP signatures: enable signing/verification on the bus when needed.

Key Paths​

  • Workflow store/engine: core/workflow/
  • Config service: core/configsvc/
  • DLQ store: core/infra/store/dlq_store.go
  • Gateway server/handlers: core/controlplane/gateway/ (thin binary: cmd/cordum-api-gateway/main.go, ships as cordum-api-gateway)
  • Safety kernel server: core/controlplane/safetykernel/ (thin binary: cmd/cordum-safety-kernel/main.go, ships as cordum-safety-kernel)
  • Scheduler/job store: core/controlplane/scheduler/, core/infra/store/job_store.go