Safety Kernel

What it decides

The Safety Kernel is Cordum's policy decision point. Two evaluation points call it for every job: the API gateway runs synchronous submit-time policy before any state is persisted or bus traffic published, and the scheduler runs pre-dispatch policy on the hot path with a 2-second timeout and a distributed circuit breaker. A sibling Output Policy Service re-evaluates results after execution, applying the same decision vocabulary to content that has already been produced.

Every evaluation returns exactly one of five decisions. Decisions compose: constraints can ride alongside an allow; denials end a run; throttles carry a Retry-After hint; approvals pause until a human signs off; and every outcome is recorded against the job so the dashboard, audit trail, and workflow engine agree on what happened.

Decision types

Decision	What happens	When returned
ALLOW	Job proceeds to dispatch (or result proceeds to the caller).	No matching rule denies, throttles, or requires approval.
DENY	Job is rejected immediately. First-class terminal status: workflow steps move to StepStatusDenied, runs to RunStatusDenied (not RunStatusFailed). on_error recovery chains still fire.	A matching rule denies the combination of tenant, topic, capability, risk tags, labels, or MCP scope.
REQUIRE_APPROVAL	Job is created in APPROVAL state without publishing to the bus. The approval is bound to the policy snapshot and job hash before the job can be requeued.	A matching rule requires human review.
THROTTLE	The Safety Kernel returns the THROTTLE decision; the gateway translates it into an HTTP 429 / gRPC ResourceExhausted response with a Retry-After header. The 429/Retry-After surface is applied at the gateway transport layer, not emitted by the kernel itself.	A velocity rule's rate budget is exhausted for the evaluated scope.
ALLOW_WITH_CONSTRAINTS	Job proceeds but with mandatory constraints (replacement topic or capability, label edits) applied before dispatch.	A matching rule allows the job but attaches non-empty constraints.

Evaluation flow

Request → Match rules → Overlay → Decision → Remediation

Rule matching

Input rules match on any combination of:

• tenants
• topics (glob match, for example job.*)
• capabilities
• risk_tags
• requires (all required entries must be present)
• pack_ids
• actor_ids
• actor_types
• labels
• secrets_present
• mcp (server / tool / resource / action)

Matched decisions normalize to allow, deny, require_approval, throttle, or allow_with_constraints. When a rule emits constraints, the response becomes ALLOW_WITH_CONSTRAINTS even if the underlying decision was allow. Approval paths set approval_required=true and bind approval_ref to the incoming job_id. Velocity rules (the rate-based family) are capped by the active licensing tier — Community gets a limited count, Team more, Enterprise unlimited; excess rules are dropped with a warning at policy load.

MCP label filtering

MCP request context is extracted from job labels:

• mcp.server | mcp_server | mcpServer
• mcp.tool | mcp_tool | mcpTool
• mcp.resource | mcp_resource | mcpResource
• mcp.action | mcp_action | mcpAction (normalized to lowercase)

MCP policy fields:

• allow_servers, deny_servers
• allow_tools, deny_tools
• allow_resources, deny_resources
• allow_actions, deny_actions

Evaluation order:

1. Rule-level match.mcp (if present)
2. Tenant-level tenants.<tenant>.mcp
3. Effective runtime safety overlay (safety.mcp)

Within each MCP field, deny takes precedence:

1. If the value is in the deny list → deny
2. Else if the allow list is non-empty and the value is not in it → deny
3. Else → allow

MCP list matching is case-insensitive exact match. Topic matching supports glob patterns; MCP fields do not.

Example:

tenants:
  default:
    mcp:
      allow_servers: ["github", "jira"]
      deny_servers: ["internal-admin"]
      allow_tools: ["search_issues", "get_issue"]
      deny_tools: ["delete_issue"]
      allow_resources: []
      deny_resources: ["repo://secret/*"]
      allow_actions: ["read", "list"]
      deny_actions: ["write", "delete"]

Policy overlay and reload

Policy is sourced from a file or URL and merged with config-service fragments. Fragment entries may declare enabled: false and are skipped when disabled; the remaining fragments are sorted by key and merged deterministically over the base policy. Each evaluation layers its own request-scoped effective-config restrictions (topics and MCP scopes) on top before deciding.

The kernel watches its source for changes. When a snapshot changes, the in-memory policy is replaced atomically and recent snapshots are tracked. Snapshot history is shared across replicas via a Redis list (cordum:safety:snapshots, LPUSH + LTRIM to 10), so the ListSnapshots RPC returns consistent results regardless of which replica handles the call. If Redis is unavailable, history falls back to a per-process in-memory list.

Decision cache

Safety decisions can be cached to keep repeat checks off the hot path. The cache key is a deterministic protobuf marshal of the PolicyCheckRequest with job_id cleared (so different jobs with the same policy-relevant input reuse the entry) and prefixed with the active snapshot hash.

Cache semantics:

• Cached responses omit approval_ref at rest. On a cache hit, approval_ref is re-bound to the current job_id when approval is required.
• Eviction first removes expired entries, then evicts the entry closest to expiration if the cache is still over capacity.
• Snapshot is part of the key, so a policy reload naturally misses old entries. Additionally, a monotonically-incrementing policy version tags every entry; when setPolicy() fires, the counter increments and the entire cache is cleared, and any entry whose tagged version does not match current is treated as a miss on lookup (belt-and-suspenders against race windows).
• In multi-replica deployments, each replica invalidates its own cache on its own policy update. No Redis coordination is involved — cache management is purely local per replica.

Policy signature verification

Policy bundles can be Ed25519-signed. Verification protects against tampering between the signer (security-ops) and the runtime loader (the Safety Kernel process): an attacker who gains write access to the policy file or its hosting URL cannot substitute rules without the signing key.

Signatures are required in production (and optionally elsewhere via configuration). The verifier accepts an inline signature, a detached signature file, or the .sig sidecar when the policy is file-based. Failure conditions include missing public key, malformed key or signature length, and verification failure — all of which refuse to load the new bundle and keep the last known-good policy in memory.

Key rotation procedure:

1. Generate a new keypair and distribute the new public key.
2. Re-sign active policy bundles with the new private key.
3. Roll the public-key configuration and signature together.
4. Keep the old key only for the rollback window; remove after cutover validation.

Remediations

Policy rules can return remediations:

• id
• title
• summary
• replacement_topic
• replacement_capability
• add_labels
• remove_labels

Remediations are returned in PolicyCheckResponse.remediations and persisted with the job's safety record.

Apply remediation endpoint:

• POST /api/v1/jobs/{id}/remediate
• Requires admin role and tenant access.
• Request body: {"remediation_id":"<id>"} (required when multiple remediations exist).

Replacement semantics in the gateway:

• A new job is cloned from the original request.
• replacement_topic overrides topic if provided.
• replacement_capability overrides meta.capability if provided.
• Labels are rewritten: add remediation_of and remediation_id, apply add_labels, then remove remove_labels.

Submit-time vs dispatch-time enforcement

Policy is evaluated at both ends of dispatch. Submit-time evaluation runs synchronously in the API gateway: deny rejects the job (HTTP 403 / gRPC PermissionDenied) with no state persisted and no bus publish; throttle returns 429 / ResourceExhausted with a Retry-After header; require-approval creates the job in APPROVAL state without publishing. These decisions are unconditional — they fire whenever the Safety Kernel returns them. Pre-dispatch evaluation runs on the scheduler's hot path and applies the same decisions before routing the job to a worker.

Denied vs Failed: denied is a distinct terminal status. Workflow runs surface StepStatusDenied → RunStatusDenied (not RunStatusFailed), and the dashboard reports denied in its own bucket. on_error recovery chains still fire for denied steps.

When the Safety Kernel is unreachable, the POLICY_CHECK_FAIL_MODE setting controls behaviour at both evaluation points:

Mode	Behavior	Risk
closed (default)	Submit-time rejects with 403; scheduler requeues with exponential backoff until the kernel recovers.	No unsafe jobs pass through; availability impact during outages.
open	Job is allowed through with a warning log and a metric increment.	Jobs bypass safety checks; use only when availability is prioritized over safety.

POLICY_CHECK_FAIL_MODE is an externally observable contract. In open mode, jobs that would normally be denied or require approval are allowed through without evaluation — use only in environments where safety violations are tolerable (e.g., staging) or where compensating controls exist downstream. Production deployments should use the default closed mode.

A Prometheus counter cordum_scheduler_input_fail_open_total (labels: topic) increments each time a job is allowed through under fail-open; alert on this counter to detect silent Safety Kernel outages.

Distributed circuit breaker

Both the input and output safety clients fail fast using a Redis-backed distributed circuit breaker. When one scheduler replica detects Safety Kernel failures, all replicas see the open circuit immediately through shared Redis state. If Redis itself is unavailable, each replica falls back to a local in-memory breaker with the same open/close thresholds. The local breaker is not permanently open — it still opens only after 3 local failures and closes after 2 local successes, exactly like the Redis-backed version. "Fail-open on Redis loss" means only that losing the shared Redis state does not itself force the safety circuit open; per-replica failure tracking continues normally, and once the local circuit opens, POLICY_CHECK_FAIL_MODE still governs whether jobs are requeued or allowed.

CLOSED --(3 failures)--> OPEN --(30s TTL expires)--> HALF_OPEN
HALF_OPEN --(2 successes)--> CLOSED
HALF_OPEN --(failure)------> OPEN

Failure recording is atomic: a single Lua script increments the shared failures counter and sets the open-duration TTL on the first failure. Open detection is a single GET; half-open transitions happen naturally when the TTL expires and Redis deletes the key; success recording DELs the key and closes the circuit. When the input circuit is open, the scheduler receives SafetyUnavailable decisions instead of blocking on RPC, and POLICY_CHECK_FAIL_MODE then decides whether to requeue or allow.

Action Gates

Beyond topic/identity policy rules, Cordum runs a deterministic pre-dispatch action-gate pipeline on the structured action descriptor. The same pipeline — tenant → file → url → mcp → mutation → provenance — runs on both the Gateway HTTP path and the Safety Kernel gRPC path, short-circuiting on the first non-allow decision and failing closed when a gate's dependency is unavailable.

Action gates consume structured fields only (never free-form prompts), source tenant identity from auth rather than the request body, and treat approval claims as untrusted until resolved against the backend approval store and audit chain. They power Cordum Edge's destructive-action enforcement.

See Action Gates for the full gate-by-gate reference.

Cross-references

• Output Policy Guide
• Action Gates
• Cordum Edge
• API Reference
• Configuration
• System Overview

Reference

Reference (source files, gRPC services, env vars, circuit-breaker tunables)

Source files

• core/controlplane/safetykernel/kernel.go
• core/controlplane/safetykernel/output_policy.go
• core/controlplane/safetykernel/scanners.go
• core/infra/config/safety_policy.go
• core/controlplane/scheduler/safety_client.go
• core/controlplane/gateway/gateway_jobs.go

gRPC services

Safety Kernel server implements:

• SafetyKernelServer — Check(), Evaluate(), Explain(), Simulate(), ListSnapshots(). Check / Evaluate / Explain / Simulate share the same evaluation path (evaluate(...) in kernel.go).
• OutputPolicyServiceServer — CheckOutput().

Wire enum note: the REQUIRE_APPROVAL decision documented above is the platform's human-facing label. On the gRPC wire, the corresponding DecisionType enum constant is DECISION_TYPE_REQUIRE_HUMAN (see core/protocol/pb/v1/pb.go). Clients decoding the protobuf PolicyCheckResponse.decision field must match against DECISION_TYPE_REQUIRE_HUMAN.

TLS for the Safety Kernel server:

• SAFETY_KERNEL_TLS_CERT
• SAFETY_KERNEL_TLS_KEY
• Production requires server TLS cert/key.
• Minimum TLS version controlled by CORDUM_TLS_MIN_VERSION (defaults to TLS 1.3 in production, TLS 1.2 otherwise).

TLS for clients (scheduler and gateway dialing the Safety Kernel):

• SAFETY_KERNEL_TLS_CA
• SAFETY_KERNEL_TLS_REQUIRED
• SAFETY_KERNEL_INSECURE (for non-production / testing)

Circuit-breaker tunables

Redis keys:

Circuit	Key Pattern	Purpose
Input safety	cordum:cb:safety:failures	Shared failure counter for SafetyClient.Check().
Output safety	cordum:cb:safety:output:failures	Shared failure counter for OutputSafetyClient.EvaluateOutput().

Constants:

Parameter	Input Safety	Output Safety
Request timeout	2s (safetyTimeout)	500ms (meta, defaultOutputMetaTimeout, override via CORDUM_OUTPUT_META_TIMEOUT_MS), 30s (content)
Open duration	30s	30s
Fail budget to open	3	3
Half-open max probes	3	3
Successes to close	2	2

In addition to the 2s SafetyClient RPC timeout above, the scheduler wraps each input safety check in a 3s defense-in-depth deadline (safetyCheckTimeout in engine.go). If the safety check exceeds 3s the engine aborts it (and POLICY_CHECK_FAIL_MODE decides whether to requeue or allow) even when the underlying RPC has not yet returned.

Wiring (in cmd/cordum-scheduler/main.go):

• SafetyClient is created with a local-only breaker, then upgraded to Redis-backed via safetyClient.WithRedis(sagaRedis).
• OutputSafetyClient uses its internal Redis connection (resultClient) for the distributed breaker automatically.

Environment variables

Variable	Component	Default	Purpose
SAFETY_KERNEL_ADDR	scheduler/gateway clients	localhost:50051	Safety Kernel gRPC address.
SAFETY_POLICY_PATH	safety kernel loader	config/safety.yaml	File policy source when URL is not set.
SAFETY_POLICY_URL	safety kernel loader	unset	URL policy source (overrides path).
SAFETY_POLICY_RELOAD_INTERVAL	safety kernel loader	30s	Policy reload interval.
SAFETY_POLICY_MAX_BYTES	safety kernel loader	2097152	Max policy size for file/URL load.
SAFETY_POLICY_URL_ALLOWLIST	safety kernel loader	unset	Comma-separated host allowlist for policy URL.
SAFETY_POLICY_URL_ALLOW_PRIVATE	safety kernel loader	false	Allow private/loopback URL hosts.
SAFETY_POLICY_CONFIG_DISABLE	safety kernel loader	unset	Disable config-service policy fragments.
SAFETY_POLICY_CONFIG_SCOPE	safety kernel loader	system	Config service scope for fragments.
SAFETY_POLICY_CONFIG_ID	safety kernel loader	policy	Config object ID for fragments.
SAFETY_POLICY_CONFIG_KEY	safety kernel loader	bundles	Config key containing policy bundle map.
SAFETY_DECISION_CACHE_TTL	safety kernel evaluator	0 (disabled)	Cache TTL for policy decisions.
SAFETY_DECISION_CACHE_MAX_SIZE	safety kernel evaluator	10000	Max cache entries before eviction.
SAFETY_POLICY_SIGNATURE_REQUIRED	safety kernel loader	true in production	Enforce signature verification.
SAFETY_POLICY_PUBLIC_KEY	safety kernel loader	unset	Ed25519 public key (base64/hex).
SAFETY_POLICY_SIGNATURE	safety kernel loader	unset	Inline signature (base64/hex).
SAFETY_POLICY_SIGNATURE_PATH	safety kernel loader	unset	Detached signature file path.
SAFETY_KERNEL_TLS_CERT	safety kernel server	unset	TLS certificate path for server listener.
SAFETY_KERNEL_TLS_KEY	safety kernel server	unset	TLS private key path for server listener.
SAFETY_KERNEL_TLS_CA	scheduler/gateway clients	unset	CA bundle for mTLS/TLS verification.
SAFETY_KERNEL_TLS_REQUIRED	scheduler/gateway clients	true in production	Require TLS when dialing safety kernel.
SAFETY_KERNEL_INSECURE	scheduler/gateway clients	false	Allow insecure client transport outside production.

Related env vars (non-SAFETY_*)

• OUTPUT_SCANNERS_PATH — scanner config file (config/output_scanners.yaml by default).
• CORDUM_ENV / CORDUM_PRODUCTION — production-mode behavior.
• CORDUM_TLS_MIN_VERSION — TLS minimum version.
• CORDUM_GRPC_REFLECTION — enable gRPC reflection.

Sign-policy helper (Go)

// sign_policy.go
// go run sign_policy.go policy.yaml private.key > policy.sig.b64
package main

import (
	"crypto/ed25519"
	"encoding/base64"
	"fmt"
	"os"
)

func main() {
	policy, _ := os.ReadFile(os.Args[1])
	priv, _ := os.ReadFile(os.Args[2]) // raw 64-byte ed25519 private key
	sig := ed25519.Sign(ed25519.PrivateKey(priv), policy)
	fmt.Println(base64.StdEncoding.EncodeToString(sig))
}