mqtt-ingestion-worker

Responsibility

services/mqtt-ingestion-worker is the single receiver for the Netmore sensor feed. It converts the external MQTT envelope into RawIngestionEventV1 and publishes that event once to the durable raw stream. It does not access the database, resolve tenant authority, or execute a telemetry flow.

Startup order

1. Validate all configuration and start the operations endpoint.
2. Connect the publisher identity to NATS and confirm the expected stream exists.
3. Connect to Netmore with MQTT 3.1.1 over TLS 1.2 or newer.
4. Resume the environment-specific persistent session with clean: false.
5. Subscribe to sensor/sensor_network_services/+/payload at QoS 1.

Readiness is successful only while both the MQTT subscription and JetStream publisher are available.

Message path

For every publish packet the worker records bounded receipt metrics, then:

1. rejects an unsupported topic, retained message, QoS, or oversized payload;
2. drops configured customer topic segments before JSON parsing;
3. preserves exact MQTT bytes and builds parsed or malformed event metadata;
4. calculates the documented idempotency key;
5. publishes JSON to sens.ingestion.raw.v1 with that key as the JetStream message ID;
6. waits for the JetStream persistence acknowledgement before allowing MQTT processing to complete.

If durable publish fails, the worker destroys the MQTT transport before PUBACK. Netmore can therefore redeliver the QoS 1 message after reconnect. This favors duplicate-safe recovery over silent loss.

Deployment and scaling

The worker is deployed with one replica and Recreate. Two active receivers using the same feed and different session state could duplicate or partition input unexpectedly. Horizontal scaling requires a new, explicitly reviewed broker/session contract.

Failure diagnosis

Readiness or metric	Meaning
MQTT disconnected	Credentials, TLS, DNS, broker reachability, or reconnect state.
Subscription inactive	Topic or QoS grant was not accepted.
JetStream unavailable	NATS auth, stream bootstrap, or NATS storage failure.
Publish failures increase	NATS persistence acknowledgement exceeded the timeout or failed.
Received grows but parsed does not	Customer filter or envelope discard rules are matching.

Use Netmore Raw Ingestion for the live runbook and the MQTT contract for exact payload semantics.