MQTT Network Server Ingestion Contract V1

Status and scope

Final for the first production raw-ingestion slice.

This contract covers Netmore sensor payload topics only. Gateway topics, gateway-change topics, downlink responses, extended topics, flow execution, measurements, and gateway-registry updates are outside V1.

Primary external references:

• Netmore MQTT
• Netmore export format

MQTT transport

Property	Contract
Broker	mqtts://mq.netmoregroup.com:8883
Protocol	MQTT 3.1.1
TLS	TLS 1.2 or newer; normal certificate verification is mandatory
Subscription	sensor/sensor_network_services/+/payload
Requested QoS	1
Session	Stable environment-specific client ID and clean=false
Keepalive	60 seconds
Reconnect	1-second exponential delay capped at 30 seconds
Max input size	256 KiB

The third topic segment is the Netmore customer. It is metadata and never a tenant authority. The configured exclusion is exact and case-normalized. A message for sens__smart_signal_gmbh is acknowledged without parsing and never reaches JetStream or PostgreSQL. No sensorType filter exists.

Retained, delayed, duplicate, out-of-order, and QoS-0 messages are accepted. QoS 0 remains at-most-once upstream. Production rollout requires observed QoS 1 in test or an explicit acceptance of the weaker guarantee.

Messages larger than 256 KiB are the only technical discard case. They are counted with a bounded reason and are not logged, queued, or stored.

Netmore All payload

The adapter accepts either one JSON object or a JSON array containing exactly one object. Unknown fields are allowed. The following fields are required:

Field	Validation and normalization
devEui	16 hexadecimal characters; normalized to uppercase
timestamp	Valid timestamp with UTC offset; normalized to UTC
payload	Non-empty, even-length hexadecimal; uppercase inside the event
fCntUp	Non-negative integer
fPort	Numeric string; normalized without leading zeroes
messageType	Literal payload
sensorType	Non-empty string

The verbose All format may additionally supply batteryLevel, freq, dr, spreadingFactor, rssi, and snr. Numeric strings and JSON numbers are accepted and normalized to finite JSON numbers. These optional values plus messageType, sensorType, and fPort are retained as Network Server metadata for Flow execution. An invalid optional connection value is omitted without discarding an otherwise valid sensor payload.

Arrays with zero or multiple elements, invalid UTF-8, invalid JSON, invalid roots, and missing or invalid required fields produce one malformed raw event. The unchanged MQTT bytes are still stored. A valid DevEUI that can be safely extracted from an otherwise malformed object may still resolve its device and tenant.

The optional Netmore gateways array is normalized into gatewayMetadata and retained with the Raw row. Gateway identifiers, RSSI, and SNR use the generic gateway metadata fields; other primitive gateway properties remain nested metadata. This is transport context only: it does not assign or update gateway registry entities. The exact source array remains available in the unchanged transport bytes.

Anonymized contract fixtures are versioned at:

• services/mqtt-ingestion-worker/test-fixtures/netmore-all-object.v1.json
• services/mqtt-ingestion-worker/test-fixtures/netmore-all-singleton-array.v1.json

They contain no production credential, customer, DevEUI, gateway identifier, MAC address, timestamp, frame counter, or sensor payload.

Idempotency

Netmore does not document a stable application message ID for this feed. A parsed event therefore uses:

netmore:v1:sha256(JSON([
  normalizedDevEui,
  normalizedUtcTimestamp,
  fCntUp,
  normalizedFPort,
  normalizedDevicePayload
]))

A malformed event uses SHA-256 over topic + NUL + exact MQTT bytes, with the same netmore:v1:sha256: prefix.

Customer, topic, gateway fields, MQTT packet identifier, MQTT DUP flag, JSON field order, and object-versus-singleton-array shape do not change the parsed business key. Timestamp, frame counter, or device payload changes do.

The receiver sets this key as Nats-Msg-Id. JetStream suppresses matching publishes for two hours. PostgreSQL raw_ingestion_idempotency_keys.idempotency_key is the final idempotency boundary and remains for 49 days.

RawIngestionEventV1

The authoritative TypeScript and runtime schema lives in @sens-platform/contracts; the matching OpenAPI component lives in docs/docs/src/openapi/platform-api.v1.yaml.

Required semantics:

• schemaVersion is raw-ingestion.v1.
• tenantId is null at the receiver.
• mqtt contains topic, QoS, retained, duplicate, local receive time, and an optional packet identifier for diagnostics only.
• transportPayload is the exact MQTT byte sequence encoded as Base64. It may represent an empty byte sequence.
• parseStatus is parsed or malformed.
• parseErrorCode is a stable bounded code and never raw input.
• devEui and rawPayload are present for parsed messages. rawPayload is only the normalized sensor payload in hex.
• networkServer contains provider netmore, service provider sensor_network_services, topic customer, format all, and optional normalized Netmore timestamp, frame counter, connection metadata, and gateway metadata.

One MQTT publish creates at most one event and at most one raw row.

Tenant resolution

raw-storage-worker performs a trusted global, case-insensitive DevEUI lookup. Only the device-registry row supplies tenant_id and device_id. A topic customer cannot create, override, or infer tenant scope.

• known parsed device: stored;
• unknown parsed device: unknown_device, with null tenant and device;
• malformed event: malformed, optionally with safely resolved tenant/device;
• repeated idempotency key: no second row and outcome duplicate.

Every distinct input is stored before retention applies. Retention is based on local mqtt.receivedAt, not the Netmore timestamp:

• a row whose DevEUI has no current platform-registry match expires after 7 days, including malformed rows without a usable DevEUI;
• every other row expires after 30 days;
• registration is evaluated again during cleanup, so a Device registered before the 7-day cleanup keeps its surviving earlier rows until the 30-day cutoff.

Administrative inspection contract

The platform API exposes two read-only, platform-admin-only operations:

• GET /v1/raw-ingestion-messages returns a reverse-chronological cursor page for a required/defaulted, maximum 42-day interval. It supports exact status, DevEUI, Netmore customer, trusted tenant, and unassigned filters. List items contain payload sizes but never payload bodies.
• GET /v1/raw-ingestion-messages/{rawIngestionMessageId}?receivedAt=... returns one exact row including the Base64 transport body and optional sensor payload. Both key parts are required because the raw table is a hypertable.

Both operations emit Cache-Control: no-store. The web console fetches the detail only when a platform administrator expands a list row. Tenant and device filters always use registry-derived columns; the Netmore customer remains external diagnostic metadata.