Netmore Raw Ingestion

Topology

Netmore MQTT
  -> mqtt-ingestion-worker
  -> NATS JetStream SENS_RAW_INGESTION_V1
  -> raw-storage-worker / raw-storage-v1
  -> TimescaleDB raw_ingestion_messages

The receiver is a singleton Deployment with Recreate strategy. The raw-storage consumer is horizontally safe because all instances share one durable pull consumer and PostgreSQL is the final idempotency boundary. NATS is cluster-internal, has no Ingress, and stores JetStream files on its own PVC.

Test and production intentionally subscribe to the full live feed. They use the same rotated Netmore portal username and password but distinct stable client IDs and separate NATS and TimescaleDB instances.

Runtime configuration

MQTT receiver

Variable	Secret	Default / requirement
SENS_MQTT_INGESTION_HTTP_HOST	no	local 127.0.0.1; Kubernetes 0.0.0.0
SENS_MQTT_INGESTION_HTTP_PORT	no	3001
SENS_NETMORE_MQTT_URL	no	mqtts://mq.netmoregroup.com:8883
SENS_NETMORE_MQTT_TOPIC	no	exact sensor topic with + customer wildcard
SENS_NETMORE_MQTT_USERNAME	yes	required; shared across test and production
SENS_NETMORE_MQTT_PASSWORD	yes	required; rotate before first live use
SENS_NETMORE_MQTT_CLIENT_ID	yes	required; stable and unique per environment
SENS_NETMORE_EXCLUDED_CUSTOMERS	no	required comma-separated exact topic segments
SENS_NETMORE_MAX_PAYLOAD_BYTES	no	262144; cannot be raised by configuration
SENS_NATS_URL	no	required cluster-internal URL
SENS_NATS_USERNAME	yes	publisher identity
SENS_NATS_PASSWORD	yes	publisher identity
SENS_NATS_PUBLISH_TIMEOUT_MS	no	5000

Raw-storage consumer

Variable	Secret	Default / requirement
SENS_RAW_STORAGE_HTTP_HOST	no	local 127.0.0.1; Kubernetes 0.0.0.0
SENS_RAW_STORAGE_HTTP_PORT	no	3002
SENS_DATABASE_URL	yes	required PostgreSQL URL
SENS_NATS_URL	no	required cluster-internal URL
SENS_NATS_USERNAME	yes	consumer identity
SENS_NATS_PASSWORD	yes	consumer identity
SENS_RAW_CONSUMER_CONCURRENCY	no	4, bounded to 1–16

Raw-retention job

Variable	Secret	Default / requirement
SENS_DATABASE_URL	yes	required PostgreSQL URL
SENS_RAW_RETENTION_DAYS	no	30, registered-device raw retention
SENS_RAW_UNREGISTERED_DEVICE_RETENTION_HOURS	no	168, seven-day unregistered retention

Helm commits only non-secret values and Secret references. Environment-specific Kubernetes Secrets provide Netmore, NATS, client-ID, and database values. Reloader annotations restart the affected Deployment or StatefulSet after rotation. A checksum on the NATS Pod template separately restarts NATS whenever its rendered ConfigMap changes, because changing the mounted file does not reload the already running server process. The shared Netmore password must be changed in both environment secret stores as one coordinated operation.

The Netmore password disclosed during planning is compromised by definition. Never place it in Git, Helm, fixtures, logs, command history, or chat again. It must be revoked and replaced before either receiver is allowed to connect.

External activation checklist

The product repository contains all code, Helm defaults, image definitions and Secret references, but it cannot create live credentials or change the separate GitOps repository. Before deployment, complete the following external work.

OpenBao

Use the Ops portal at /secrets and populate these catalog groups in both environment mounts:

Deployment	Mount	Required groups
Test	sens-test	runtime/netmore-mqtt, runtime/raw-storage-worker, runtime/nats-auth
Production	sens-production	runtime/netmore-mqtt, runtime/raw-storage-worker, runtime/nats-auth

Create these exact keys:

Folder	Key	Value rule
/netmore-mqtt	SENS_NETMORE_MQTT_USERNAME	newly valid Netmore user; same value in both projects
/netmore-mqtt	SENS_NETMORE_MQTT_PASSWORD	rotated outside Git and chat; same new value in both projects
/netmore-mqtt	SENS_NETMORE_MQTT_CLIENT_ID	stable, allowed characters only, and different in test and production
/raw-storage-worker	SENS_DATABASE_URL	environment-local PostgreSQL URL for sens_platform; never point test at production
/nats-auth	bootstrap-username	unique environment-local bootstrap identity
/nats-auth	bootstrap-password	independent high-entropy password
/nats-auth	mqtt-username	unique environment-local MQTT publisher identity
/nats-auth	mqtt-password	independent high-entropy password
/nats-auth	raw-storage-username	unique environment-local raw consumer identity
/nats-auth	raw-storage-password	independent high-entropy password

The historical deployed chart still consumes publisher-* and consumer-*. Keep those four legacy keys alongside the new names until the coordinated chart and release-values upgrade has been verified in test and production.

Do not enter SENS_NETMORE_EXCLUDED_CUSTOMERS in OpenBao. Helm already sets the non-secret exact exclusion to sens__smart_signal_gmbh. No new frontend or platform-api secret is needed for the raw-data view.

External GitOps repository

In sens-platform-infra, add three ExternalSecret resources per environment that map the folders above to these namespace-local targets:

Namespace	Targets
sens-test	sens-test-netmore-mqtt-runtime, sens-test-raw-storage-runtime, sens-test-nats-auth
sens-production	sens-production-netmore-mqtt-runtime, sens-production-raw-storage-runtime, sens-production-nats-auth

Then copy the ingestion sections from this chart’s environment values into the external environment overrides: embedded NATS, both workers, stream bootstrap, retention Job, Secret references, Reloader annotations and synchronization waves. Add immutable release tags for mqtt-ingestion-worker, raw-storage-worker, raw-ingestion-bootstrap and raw-ingestion-retention; the bootstrap uses the raw-storage-worker image and retention uses the database-migrations image. The web-console and platform-api tags must also select the same release that contains the raw-data view and its API.

The checked-in product values enable services.webConsole in Production at https://app.iot-sens.schlossers.at with API base URL https://api.iot-sens.schlossers.at. Before promoting that product revision, the private sens-platform-infra repository must make the matching coordinated change:

1. configure services.webConsole in the Production static values with the exact host, API base URL, Traefik class, letsencrypt-prod-sens issuer, and TLS Secret sens-platform-web-console-production-tls, while keeping the static activation gate disabled;
2. add https://app.iot-sens.schlossers.at to the exact Production API CORS allowlist without adding the test origin or a wildcard;
3. include webConsoleTag in the Production promotion workflow’s image keys and generated release.yaml, and enable the web-console with that exact tag in generated release.values.yaml;
4. merge the infrastructure preparation before running the product Production promotion, then review and synchronize the resulting Application diff manually.

The hostname, runtime URL, CORS origin, and TLS Secret reference are deployment configuration, not OpenBao secrets. The web-console itself has no runtime Secret or OpenBao group.

The TimescaleDB and GHCR bootstrap values now have catalog groups and synchronize to the existing Secret names. Verify readiness without decoding them during a normal rollout.

Safe verification

After synchronization, verify names and Ready conditions without printing values:

sudo kubectl -n sens-test get externalsecret
sudo kubectl -n sens-test get secret sens-test-netmore-mqtt-runtime sens-test-raw-storage-runtime sens-test-nats-auth
sudo kubectl -n sens-test rollout status statefulset/sens-platform-test-nats
sudo kubectl -n sens-test rollout status deployment/sens-platform-test-raw-storage-worker
sudo kubectl -n sens-test rollout status deployment/sens-platform-test-mqtt-ingestion-worker

Repeat in production only after the 24-hour test gate and capacity projection. An existing Secret proves only synchronization, not correctness; the receiver and consumer readiness endpoints are the final dependency check.

Administrative raw-data view

The web console exposes Raw-Daten only to platform administrators. Its list request is limited to at most 42 days for API compatibility, while retained raw data is bounded to 30 days for registered Devices and 7 days for unknown or malformed input. Exact MQTT bytes and sensor payload are fetched lazily when a row is expanded. The API uses Cache-Control: no-store; Netmore customer values never grant tenant authority, and a selected tenant filters only the trusted device-registry assignment. The authoritative HTTP surface is versioned in the platform OpenAPI document.

Operational investigations load 25 metadata rows at a time. The first request fixes the exclusive to anchor; the cursor binds that anchor, scope, and all filters to later pages. Refresh starts with a new anchor. There is no exact timeline total. Equal receive timestamps use the Raw-message UUID as the unique descending tie-breaker. Continuation failures preserve loaded rows and remain retryable; repeated or late cursor responses cannot append evidence twice.

JetStream contract

Setting	Value
Server image	nats:2.14.0-alpine3.22
Stream	SENS_RAW_INGESTION_V1
Subject	sens.ingestion.raw.v1
Retention	Work queue
Storage	File
MaxAge	7 days for unacknowledged backlog
MaxBytes	2 GiB
MaxMsgSize	512 KiB
Discard	New
Duplicate window	2 hours
Replicas	1
Server file store	3.8 GB
PVC	4 GiB test/default; 5 GiB production
Consumer	raw-storage-v1
Ack policy	Explicit
MaxAckPending	16
MaxDeliver	100
Pull batch	16
Backoff	1s, 5s, 30s, 2m, 10m, 30m, then 1h

The bootstrap source declares a 60-second AckWait. NATS applies the first configured backoff value as the effective acknowledgement timeout and reports one second in consumer info. The worker sends a progress acknowledgement every 500 milliseconds while the database transaction is active and uses explicit delayed NAK values from the same schedule. Revisit this NATS semantic before adding other slow processing inside the consumer.

An acknowledgement confirmed after the PostgreSQL commit removes the message from the work queue. MaxAge and MaxBytes therefore bound only unprocessed outage backlog; neither setting defines raw-data retention. If PostgreSQL is unavailable, the worker sends no successful acknowledgement and JetStream keeps the message for retry.

The versioned bootstrap Job creates missing resources, updates compatible limits, and rejects incompatible stream retention, storage, discard, replica, subject, or pull-consumer acknowledgement configuration. Because NATS retention is immutable, an existing limits-retention stream must be drained, verified, and recreated once; the bootstrap never deletes it automatically. Bootstrap, publisher, and consumer use separate NATS users.

Retention

The Apache-2 TimescaleDB image supports manual drop_chunks but not automatic retention policies. An hourly CronJob runs at minute 17 with concurrencyPolicy: Forbid:

1. drop raw chunks fully older than 30 days;
2. delete rows older than 7 days when their DevEUI is not currently in the platform registry, including malformed rows without a DevEUI;
3. delete any remaining rows older than 30 days in bounded batches;
4. delete idempotency keys whose 49-day expiry has passed;
5. emit one structured summary with duration, dropped-chunk count, deleted-row counts, and deleted-key count.

Registration is evaluated again during cleanup. A Device registered before the 7-day cleanup therefore keeps its surviving earlier rows until the 30-day cutoff. With the hourly schedule, eligible rows are normally removed within one hour after their cutoff. The Job can be suspended in Helm during an incident or data investigation.

Local integration profile

The normal local stack remains database-only. The explicit ingestion profile adds NATS, migrations, bootstrap, raw storage, and the receiver:

docker compose --project-directory . \
  -f infra/docker/compose.yaml \
  --profile ingestion up --build

The receiver has no local Netmore defaults. Without deliberately supplied SENS_NETMORE_MQTT_USERNAME, SENS_NETMORE_MQTT_PASSWORD, and a stable SENS_NETMORE_MQTT_CLIENT_ID, its configuration validation fails before an external connection is attempted. There is no permanent mock MQTT broker.

The gated real-NATS/TimescaleDB integration test uses short-lived or dedicated test infrastructure and these variables:

SENS_RUN_INGESTION_INTEGRATION_TESTS=true
SENS_TEST_DATABASE_URL
SENS_TEST_NATS_URL
SENS_TEST_NATS_BOOTSTRAP_USERNAME / PASSWORD
SENS_TEST_NATS_PUBLISHER_USERNAME / PASSWORD
SENS_TEST_NATS_CONSUMER_USERNAME / PASSWORD

Rollout

1. Revoke the disclosed Netmore password and generate the replacement outside the repository and chat.
2. Store the shared user/password and distinct stable client IDs in the two environment secret stores.
3. Add both worker images and Secret mappings to the external GitOps values.
4. Stop the test MQTT receiver and wait until the old raw consumer reports num_pending=0, num_ack_pending=0, and a stream acknowledgement floor equal to the stream’s last sequence. Stop the raw consumer after this proof. Abort the migration if any check differs.
5. Verify that the managed test NATS StatefulSet retains its original 2-GiB volume-claim template while its expanded bound PVC reports 5 GiB. Production retains its existing 5-GiB template. The chart default for new installations is 4 GiB, but an existing StatefulSet volume-claim template is immutable and is not rewritten when its PVC grows. If another real resize or reprovisioning is required, preserve or drain every other JetStream stream on that PVC as a separate controlled operation.
6. Delete only the verified empty legacy SENS_RAW_INGESTION_V1 stream with the bootstrap identity. Do not delete raw database rows. Deploy NATS and run the new bootstrap so it recreates the stream with work-queue retention.
7. In test, deploy the database image, raw consumer, retention CronJob, and receiver in that order.
8. Observe test for at least 24 hours. Confirm incoming QoS 1, object/array shape, retained behavior, exclusion, stable lag, status distribution, and storage size.
9. Simulate a database outage. Confirm backlog growth, recovery, and exactly one row per idempotency key.
10. Project the 30-day registered and 7-day unregistered table, index, and payload size. Production activation requires the projection plus reserve to remain below 70% of the 10-GiB PVC; otherwise enlarge the PVC first.
11. Repeat the drain, legacy-stream recreation, activation with its own client ID, dependency checks, and write verification in production. The existing 5-GiB production PVC remains in place.

At the assumed maximum of 30 accepted messages per minute, each environment stores about 43,200 rows per day. A separate low-cardinality counter must show the high-volume exclusion, while queue and database queries must show zero rows for that customer.

Recovery and rollback

• Disable mqtt-ingestion-worker first, then raw-storage-worker.
• Do not delete the NATS StatefulSet PVC or raw tables.
• Suspend the retention CronJob when data correctness is uncertain.
• Do not down-migrate the additive raw schema in production. Apply a forward fix; malformed rows intentionally make the development down path refuse a destructive rollback.
• A successful database commit followed by a lost consumer acknowledgement is safe: redelivery reaches the PostgreSQL idempotency key and creates no row.
• Invalid internal event contracts are terminated and counted without payload logging.
• Max-delivery events remain unacknowledged in the work queue until MaxAge or MaxBytes removes them. After repair, reset the existing durable consumer according to the recovery procedure. There is no DLQ stream.
• JetStream is not a database backup.
• An MQTT/NATS outage beyond Netmore’s documented two-hour durable-subscriber window is an explicit upstream data-loss risk.

Operational verification

Receiver readiness requires active MQTT connection, confirmed subscription, and the expected JetStream stream. Consumer readiness requires PostgreSQL and the expected durable consumer. Health is process liveness only.

Metrics intentionally exclude topic, customer, tenant, device, DevEUI, and payload labels. Logs contain event/correlation IDs, bounded status codes, and dependency context only. See Observability Model for the metric inventory.