Functional-Safety Framing#
Much of the current AI-safety discussion is reinventing concepts that functional safety engineers solved decades ago. ANNIE starts from a different premise.
An autonomous AI agent that dispatches actions to the real world is, in functional-safety terms, a high-capability but unreliable probabilistic input channel feeding a safety-related control system. The field already has mature, battle-tested standards for exactly this situation.
What this means#
For the past three decades, machinery, automotive systems, medical devices, and process plants have been designed against the IEC 61508 family of functional-safety standards:
- IEC 61508 — the parent standard. Defines Safety Integrity Levels (SIL 1–4).
- ISO 13849-1 — safety-related parts of machine control systems. Defines Performance Levels (PL a–e) and Categories (B, 1, 2, 3, 4).
- IEC 62061 — machine-safety SIL specifically.
- ISO 26262 — automotive (ASIL A–D).
The common thread across all of them: you do not trust a single probabilistic component to keep a system safe. You build a topology — a structural arrangement of channels, monitors, and diagnostics — such that the safety function survives any single fault, including a fault in your most expensive component.
ANNIE applies this body of knowledge to the problem of autonomous AI agent execution. The language model is treated as the unreliable, high-capability channel. The deterministic safety kernel is the independent monitor. The architectural patterns are the same ones that keep elevators from dropping people, autonomous emergency braking from latching on, and modern flight-control computers from acting on a corrupted sensor reading.
A note on claims#
The phrasing here is deliberate and worth understanding:
- ANNIE is designed against the functional-safety patterns described below.
- ANNIE achieves architectural alignment with the dual-channel topology specified by ISO 13849-1 Category 3 / 4.
- ANNIE is not ISO 13849-1 certified. It has not been assessed by a Notified Body, does not carry CE / UKCA marking, and does not publish MTTFd, DC, or CCF figures against the standard’s methodology.
Certification under any of these standards requires third-party assessment, channel-level reliability data, diagnostic-coverage calculations, common-cause-failure avoidance scoring, and a documented safety case. None of that is in scope for ANNIE today. The body of standards is used here as a design framework, not a compliance label. We will not pretend otherwise.
The architectural alignment is real. The certificate is a separate workstream. Building with this vocabulary from the start is meant to make eventual review faster and clearer: auditors, carriers, and safety engineers can map ANNIE’s threat model onto concepts they already use for dangerous equipment.
How the categories map onto AI agents#
| ISO 13849-1 Category | Industrial example | AI-agent realization |
|---|---|---|
| B / 1 — basic, well-tried single channel | One switch wired to one contactor | A bare prompt directly calling tools. One injection or hallucination collapses the safety function. |
| 2 — single channel with periodic test | Output filter or moderator running after generation, on a fixed interval | The hazard can fire in the gap between checks. |
| 3 — dual-channel; no single fault loses the safety function | Redundant channels with cross-monitoring | A probabilistic channel proposes; a deterministic channel (formal logic, schema-checked) must sign before commit. |
| 4 — dual channel with continuous diagnostic, faster than the hazard rate | Continuous self-monitoring inside the dual channel | The same dual channel plus a continuous safety-kernel cycle that fires faster than any actuation can complete. |
ANNIE’s architecture is designed toward the Category 4 pattern. The deterministic-cadence control loop runs the continuous diagnostic. The hardware-fault response thread performs the safe-state-on-fault transition within a documented microsecond budget on supported hardware. The hash-chained append-only event log together with the fixed-size cryptographic memory witnesses provide the safe-state recovery surface. The agent’s language model is the high-capability but presumed-untrusted probabilistic channel; the formal-logic kernel is the independent monitor.
Category 4 in the standard’s strict sense requires very high diagnostic coverage (≥99%) and fault-detection time shorter than the process safety time, validated through a documented safety case. We use “designed toward” deliberately: the architectural pattern is present and engineered for; the validation paperwork that would let us claim Category 4 outright is a future workstream. The benefit today is audit readiness: hazards, diagnostics, timing evidence, and replay surfaces are shaped for eventual third-party review instead of being reconstructed after deployment.
Why this framing matters now#
The regulatory landscape for AI is converging on exactly this body of standards. Several reference points:
- ISO/IEC TR 5469:2024 — a published technical report on AI and functional safety.
- ISO/IEC 23894:2023 — AI risk management.
- ISO/IEC 42001:2023 — AI management system.
- NIST AI RMF 1.0 — the US-side risk-management framework.
- EU AI Act — high-risk AI systems will need evidence of compliance as obligations come into application, and harmonized standards work is converging around safety and risk-management evidence.
As the EU AI Act’s harmonized standards mature, platforms designed with these patterns from day one should have a structural advantage over those attempting retrofits. A vendor that built its AI agent platform against functional-safety topology early is in a stronger position than one assembling a compliance story after the fact.
What this is not#
This framing does not say AI agents are machines. It does not say a language model is a contactor. It says that the problem of safely executing actions proposed by an unreliable high-capability component is structurally similar to the problem industrial-safety engineering has been solving for a generation, and the structural patterns transfer.
That is why the language matters. When autonomous agents are described with the same threat-model vocabulary used for dangerous equipment, the risk becomes recognizable: unsafe command, independent monitor, diagnostic coverage, safe state, fault-detection time, replayable evidence. The analogy is not a certification claim; it is a translation layer that lets reviewers reason about a new technology with tools they already trust.
If your worldview is that AI is sui generis and needs a brand-new safety paradigm, you may find this framing reductive. If your worldview is that we should use vocabulary that exists rather than inventing parallel vocabulary, you will find it efficient.
Related#
- The Iron Lung — the continuous diagnostic that makes the Category 4 pattern actually hold.
- The Guillotine — the safe-state-on-fault response.
- The Ledger of Reality — the safe-state recovery mechanism.
- Holographic Tombstones — bounded memory under the diagnostic budget.