ADR-0008 Fail-Closed Audit and Break-Glass Administration

Status

Accepted on 2026-08-03.

Context

Administrative access must answer who did what and when, including failed API calls and reads. Logging only after a successful handler can hide calls during process failure and allows an audit outage to leave sensitive operations unrecorded. Global platform administration also needs a recoverable account without allowing concurrent requests to remove every administrator.

Decision

Every known /v1 route must declare audit metadata. A Fastify registration assertion rejects any future route that omits it. Before body parsing, authentication, authorization, or handler execution, platform-api inserts an api_request row with outcome incomplete. Failure to insert returns 503 and prevents handler execution.

Authentication enriches the row with actor and session. Tenant scope is added only after platform-admin or tenant authorization succeeds. Before the response is sent, the row is finalized with the route template, status, outcome, end time, and duration. A finalization or process failure leaves the incomplete row as evidence. Entity mutations write a separate entity_change event in the same transaction as the entity update.

Global and tenant audit APIs use bounded reverse cursor queries. The default range is 24 hours, the maximum explicit range is 31 days, and retention is seven calendar years. Tenant queries return only safely scoped rows. Bodies, query values, credentials, headers, IP addresses, and user-agent contents are never stored.

Every account actually created by auth:admin receives the permanent database marker admin_cli_bootstrap. Existing bootstrap audit events are backfilled, and an audit trigger also marks accounts created by the previous CLI image after an application rollback. Protected accounts cannot be disabled, deleted, or stripped of platform_admin through HTTP; profile and password changes remain possible.

Platform-role grant, revoke, and user-disable operations share a PostgreSQL transaction advisory lock. They additionally reject self-lockout and removal of the last active platform administrator.

Consequences

• PostgreSQL and audit persistence are mandatory for every /v1 request.
• Each request adds one insert and one update; mutations add a transactional entity-audit insert.
• Audit reads audit themselves but exclude their own current incomplete row.
• Operational and unknown paths remain outside the audit contract.
• The additive schema remains readable by the previous API image.
• Break-glass protection is intentionally stronger than ordinary last-admin protection and cannot be removed through HTTP.

Rejected alternatives

• Best-effort post-response audit was rejected because outages and crashes would silently erase evidence.
• Auditing raw URLs, bodies, query strings, IP addresses, or user agents was rejected because it increases credential, PII, and tenant-data exposure.
• A minimum tenant-admin count was rejected because platform administrators are the recovery authority for tenants.
• In-memory or instance-local admin locks were rejected because they are unsafe under horizontal scaling.

Rollout and rollback

The migration runs before the new API image. It uses defaults for all new audit columns and does not change queries required by the previous image. Production rollback selects the previous image and keeps the schema. The old CLI continues to create protected accounts through the compatibility trigger. Schema defects are corrected forward rather than by a production down migration.