Device Ingestion Architecture
Purpose
Section titled “Purpose”This document defines the initial ingestion architecture for LoRaWAN telemetry messages delivered by the external Network Server over MQTT.
Initial Source
Section titled “Initial Source”Network Server: Netmore.
The Network Server decrypts LoRaWAN data and publishes messages over MQTT.
High-Level Flow
Section titled “High-Level Flow”Network Server -> MQTT -> mqtt-ingestion-worker -> internal stream -> telemetry-worker -> database
mqtt-ingestion-worker Responsibilities
Section titled “mqtt-ingestion-worker Responsibilities”The MQTT ingestion worker must:
- Connect to the configured MQTT broker.
- Subscribe to configured topics.
- Reconnect safely on disconnect.
- Validate the minimal message envelope.
- Attach ingestion metadata.
- Publish messages to the internal stream.
- Emit metrics and structured logs.
- Avoid executing decoders.
- Avoid direct long-term telemetry writes where practical.
telemetry-worker Responsibilities
Section titled “telemetry-worker Responsibilities”The telemetry worker must:
- Consume raw ingestion events.
- Deduplicate messages where possible.
- Match messages to devices.
- Store raw payloads.
- Run the configured decoder.
- Store normalized measurements.
- Record decoder failures.
- Record unknown-device failures.
- Handle delayed and out-of-order messages.
- Send failed messages to a dead-letter flow when needed.
Delivery Assumptions
Section titled “Delivery Assumptions”The platform must tolerate:
- Duplicate messages.
- Out-of-order messages.
- Delayed messages.
- Malformed payloads.
- Unknown devices.
- Decoder errors.
- Temporary broker outages.
- Temporary database outages.
Idempotency
Section titled “Idempotency”The ingestion pipeline must use available identifiers for idempotency.
Possible idempotency inputs:
- Network Server message ID.
- DevEUI.
- Frame counter.
- Received timestamp.
- Payload hash.
- Source identifier.
The exact strategy must be defined after the Netmore MQTT payload is documented.
MQTT QoS
Section titled “MQTT QoS”MQTT QoS must be documented before implementation.
QoS levels:
- QoS 0 means at most once delivery.
- QoS 1 means at least once delivery and duplicates are possible.
- QoS 2 means exactly once at MQTT protocol level, but application-level idempotency is still recommended.
Unknown Devices
Section titled “Unknown Devices”Unknown device messages must not crash ingestion.
They should be:
- Stored or recorded according to retention rules.
- Counted with metrics.
- Logged safely.
- Made visible to operators where useful.
Decoder Failures
Section titled “Decoder Failures”Decoder failures must not stop the ingestion pipeline.
They should be:
- Recorded with safe error details.
- Counted with metrics.
- Linked to device, device type, decoder version, and message ID where possible.
- Available for operational troubleshooting.