ADR-0004: Internal Broker

Status

Accepted and amended on 2026-08-24.

Context

Real Netmore ingestion needs durable buffering, retry, replay, backpressure, and failure isolation. Running a broker before an ingestion capability exists adds unused resources and operational surface.

Publishing one MQTT input independently to both raw and flow queues also creates a dual-write consistency problem.

Decision

Use self-hosted NATS JetStream for the implemented first Netmore ingestion slice. It is enabled by an explicit local profile and in the test and production Helm values.

The first stage uses a durable raw stream between the MQTT receiver and the raw-storage consumer. The consumer uses explicit acknowledgements and acknowledges only after the raw database transaction commits. Work-queue retention removes acknowledged messages, so JetStream is an outage buffer and not a second raw-data archive. Unacknowledged messages remain durable until a configured stream limit is reached.

A later flow stage receives a raw-stored event produced through a transactional outbox after the raw commit. The MQTT receiver does not dual-publish to parallel streams. Raw Stored also uses work-queue retention: the Flow acknowledgement removes a processed reference immediately, and a 7-day maximum age plus a 1-GiB byte limit bounds only unprocessed backlog. Pinned replay remains a separate stream.

Final V1 resources

• NATS image: nats:2.14.0-alpine3.22;
• stream: SENS_RAW_INGESTION_V1;
• subject: sens.ingestion.raw.v1;
• file-backed work-queue retention, one replica;
• 7-day MaxAge, 2-GiB MaxBytes, 512-KiB maximum message size;
• DiscardNew, 2-hour duplicate window, 3.8-GB server file-store limit;
• 4-GiB test/default PVC and existing 5-GiB production PVC;
• durable pull consumer: raw-storage-v1;
• explicit acknowledgements, MaxAckPending=16, MaxDeliver=100;
• pull batch 16 and bounded consumer concurrency 4;
• backoff at 1 second, 5 seconds, 30 seconds, 2 minutes, 10 minutes, 30 minutes, then 1 hour.

A versioned bootstrap Job owns stream and consumer configuration. NATS does not allow an existing stream to change its retention policy. Normal bootstrap mode therefore rejects incompatible streams. A separate one-time guarded mode may replace only the known Raw Stored limits-retention stream after the Flow consumer is fully drained and the state remains unchanged for five seconds. Helm rejects that mode while MQTT ingestion, raw storage, or the Flow worker is enabled. Bootstrap, publisher, and consumer have separate credentials. There is no Raw Ingestion DLQ stream; max-delivery signals and the recovery runbook govern replay.

Alternatives

RabbitMQ remains a viable adapter replacement. Kafka is operationally heavier than required for the initial volume. Direct MQTT-to-database processing is rejected because it couples broker receipt to database availability.