Functional-Safety Framing#
The AI-safety conversation keeps inventing vocabulary for problems that industrial control engineering already solved in the 1990s. ANNIE is built on the recognition that an autonomous AI agent capable of dispatching actions to a real system is, in functional-safety terms, a probabilistic input channel feeding a control system that can cause harm. There is a body of standards purpose-built for that situation.
What this means#
For the past three decades, machinery, automotive systems, medical devices, and process plants have been designed against the IEC 61508 family of functional-safety standards:
- IEC 61508 — the parent standard. Defines Safety Integrity Levels (SIL 1–4).
- ISO 13849-1 — safety-related parts of machine control systems. Defines Performance Levels (PL a–e) and Categories (B, 1, 2, 3, 4).
- IEC 62061 — machine-safety SIL specifically.
- ISO 26262 — automotive (ASIL A–D).
The common thread across all of them: you do not trust a single probabilistic component to keep a system safe. You build a topology — a structural arrangement of channels, monitors, and diagnostics — such that the safety function survives any single fault, including a fault in your most expensive component.
ANNIE applies this body of knowledge to the problem of autonomous AI agent execution. The language model is treated as the unreliable, high-capability channel. The deterministic safety kernel is the independent monitor. The architecture is engineered against the same patterns that keep elevators from dropping people and emergency stops from latching open.
A note on claims#
The phrasing here is deliberate and worth understanding:
- ANNIE is designed against the functional-safety patterns described below.
- ANNIE achieves architectural alignment with the dual-channel topology specified by ISO 13849-1 Category 3 / 4.
- ANNIE is not ISO 13849-1 certified. It has not been assessed by a Notified Body, does not carry CE / UKCA marking, and does not publish MTTFd, DC, or CCF figures against the standard’s methodology.
Certification under any of these standards requires third-party assessment, channel-level reliability data, diagnostic-coverage calculations, common-cause-failure avoidance scoring, and a documented safety case. None of that is in scope for ANNIE today. The body of standards is used here as a design framework, not a compliance label. We will not pretend otherwise.
The architectural alignment is real. The certificate is a separate workstream.
How the categories map onto AI agents#
| ISO 13849-1 Category | Industrial example | AI-agent realization |
|---|---|---|
| B / 1 — basic, well-tried single channel | One switch wired to one contactor | A bare prompt directly calling tools. One injection or hallucination collapses the safety function. |
| 2 — single channel with periodic test | Asynchronous safety check on a fixed interval | Post-hoc output moderator. The hazard can fire in the gap between checks. |
| 3 — dual-channel; no single fault loses the safety function | Redundant channels with cross-monitoring | A probabilistic channel proposes; a deterministic channel (formal logic, schema-checked) must sign before commit. |
| 4 — dual channel with continuous diagnostic, faster than the hazard rate | Continuous self-monitoring inside the dual channel | The same dual channel plus a continuous safety-kernel cycle that fires faster than any actuation can complete. |
ANNIE’s architecture aligns with the Category 4 pattern. The Iron Lung’s deterministic control loop is the continuous diagnostic. The Guillotine is the safe-state-on-fault response. The Ledger of Reality plus Holographic Tombstones are the safe-state recovery mechanism. The agent’s language model is the high-capability but presumed-untrusted probabilistic channel; the formal-logic kernel is the independent monitor.
Why this framing matters now#
The regulatory landscape for AI is converging on exactly this body of standards. Several reference points:
- ISO/IEC TR 5469 — the in-progress technical report explicitly bridging AI and functional safety. It will eventually do the work this page is doing here, formally.
- ISO/IEC 23894:2023 — AI risk management.
- ISO/IEC 42001:2023 — AI management system.
- NIST AI RMF 1.0 — the US-side risk-management framework.
- EU AI Act — high-risk AI systems will need evidence of compliance, and the harmonized standards being drafted explicitly cross-reference the IEC 61508 family.
A vendor that built its AI agent platform against functional-safety topology before the standards were written is in a substantially stronger position than one retrofitting after.
What this is not#
This framing does not say AI agents are machines. It does not say a language model is a contactor. It says that the problem of safely executing actions proposed by an unreliable high-capability component is structurally similar to the problem industrial-safety engineering has been solving for a generation, and the structural patterns transfer.
If your worldview is that AI is sui generis and needs a brand-new safety paradigm, you may find this framing reductive. If your worldview is that we should use vocabulary that exists rather than inventing parallel vocabulary, you will find it efficient.
Related#
- The Iron Lung — the continuous diagnostic that makes the Category 4 pattern actually hold.
- The Guillotine — the safe-state-on-fault response.
- The Ledger of Reality — the safe-state recovery mechanism.
- Holographic Tombstones — bounded memory under the diagnostic budget.