Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

ADR-006: Circuit Breaker on Safety Kernel Client

  • Status: Accepted
  • Date: 2026-01-15

Context​

The Safety Kernel is a synchronous dependency on the scheduler hot path (see ADR-001). If the Safety Kernel becomes unavailable or slow, the scheduler cannot dispatch any jobs. Without a fallback mechanism, a Safety Kernel outage causes total platform unavailability.

Decision​

The scheduler's Safety Kernel client uses a circuit breaker pattern with configurable thresholds.

States​

CLOSED (normal) → OPEN (tripped) → HALF-OPEN (probe)
        ↑                                    │
        └────────────────────────────────────┘
  • Closed: All requests forwarded to Safety Kernel normally.
  • Open: After N consecutive failures, the circuit trips. Requests are short-circuited with a fallback decision (configurable: deny-all or allow-all).
  • Half-Open: After a cooldown period, one probe request is sent. If it succeeds, the circuit closes. If it fails, the circuit reopens.

Configuration​

ParameterDefaultDescription
Failure threshold3Consecutive failures before tripping (safetyCircuitFailBudget)
Cooldown period30sTime in open state before probing (safetyCircuitOpenFor)
Half-open probes3Probe budget while half-open (safetyCircuitHalfOpenMax)
Successes to close2Consecutive probe successes that re-close the circuit (safetyCircuitCloseAfter)
Timeout per call2sgRPC deadline for safety-kernel calls (safetyTimeout)

Behavior When the Circuit Is Open​

The circuit breaker itself does not decide allow/deny. When the circuit is open, SafetyClient.Check returns a SafetyUnavailable record (reason "safety kernel circuit open"); the scheduler then resolves what to do with the job using its fail mode:

  • fail-closed (default): the job is not allowed through — the scheduler requeues it via RetryAfter until the kernel recovers. No unchecked action executes. Set with WithInputFailMode("closed").
  • fail-open (opt-in): the job is dispatched with a SafetyAllow decision, tagged safety_bypassed=true, and a dedicated safety-bypass audit event is emitted. Set with WithInputFailMode("open").

Fail mode is resolved per request: a per-tenant FailModeResolver overrides the global flag when configured (isInputFailOpenForTenant), otherwise the global default (fail-closed) applies. The same model governs the async output-safety check (isAsyncFailOpenForTenant).

Observability​

  • IncSafetyUnavailable(topic) — incremented whenever the kernel is unavailable (circuit open or call error).
  • IncInputFailOpen(topic) — incremented when a job is allowed through under fail-open.
  • Redis counter cordum:scheduler:input_fail_open_total — durable fail-open tally.
  • A safety-bypass audit event is emitted on every fail-open dispatch so SIEM can detect bypass-on-unavailable without parsing logs.

Key source files:

  • core/controlplane/scheduler/circuit_breaker.go — RedisCircuitBreaker (distributed breaker + local fallback)
  • core/controlplane/scheduler/safety_client.go — breaker thresholds, safetyTimeout, open-circuit handling
  • core/controlplane/scheduler/engine.go — SafetyUnavailable handling, fail-open/fail-closed resolution

Consequences​

Positive:

  • Platform degrades gracefully instead of hanging on Safety Kernel outage
  • Configurable fail mode (per-tenant or global) lets operators choose safety vs availability tradeoff
  • Automatic recovery when Safety Kernel comes back (no manual intervention)
  • Observability signals (fail-open metrics + audit events) enable alerting on bypass-on-unavailable

Tradeoffs:

  • fail-closed (default) means jobs are requeued and processing stalls during an outage
  • fail-open means unchecked actions can execute (risk) — gated behind explicit opt-in + audit trail
  • Circuit breaker adds complexity to the evaluation path
  • Probe requests during half-open may see stale results