Raw Stored Work-Queue Migration

Purpose

This runbook converts the existing SENS_RAW_STORED_V1 stream from limits retention to work-queue retention. The stream name, subject sens.ingestion.raw-stored.v1, and durable consumer flow-worker-v1 do not change. A confirmed Flow acknowledgement removes its reference immediately; unacknowledged backlog is bounded by a 7-day MaxAge and 1-GiB MaxBytes.

NATS cannot change a stream retention policy in place. The one-time bootstrap mode therefore deletes and recreates only the verified legacy stream. Run Test first, observe it for at least 24 hours, and only then repeat the same process in Production.

Safety properties

The migration command accepts only the exact known legacy contract: one sens.ingestion.raw-stored.v1 subject, limits retention, file storage, DiscardNew, and one replica. It requires exactly one attached consumer, flow-worker-v1, and that consumer must report:

• num_pending=0;
• num_ack_pending=0;
• identical delivered and acknowledgement-floor consumer sequences;
• identical delivered and acknowledgement-floor stream sequences.

It reads the stream and consumer again after five seconds. A changed stream sequence, contract, pending count, acknowledgement count, or consumer sequence aborts without deletion. Missing, unexpected, or partially drained resources also abort without deletion. Logs contain only resource names, bounded counters, sequences, and the outcome.

The Helm chart enforces the maintenance topology. Setting rawIngestionBootstrap.migrateRawStoredRetention=true fails rendering unless the bootstrap is enabled and all of these workloads are disabled:

• services.mqttIngestion;
• services.rawStorage;
• services.flowWorker.

Database-capacity gate

The SQL lifecycle changes to 30 days for registered Devices and 7 days for currently unknown or malformed rows. Do not change the TimescaleDB PVC blindly. Use at least seven complete days of representative input and collect:

select hypertable_size('raw_ingestion_messages') as raw_bytes,
       pg_database_size(current_database()) as database_bytes;

select date_trunc('day', received_at) as day,
       count(*) filter (
         where dev_eui is not null
           and exists (
             select 1 from devices
             where upper(devices.dev_eui) = upper(raw_ingestion_messages.dev_eui)
           )
       ) as registered_rows,
       count(*) filter (
         where dev_eui is null
            or not exists (
              select 1 from devices
              where upper(devices.dev_eui) = upper(raw_ingestion_messages.dev_eui)
            )
       ) as short_lifecycle_rows
from raw_ingestion_messages
where received_at >= date_trunc('day', now()) - interval '7 days'
  and received_at < date_trunc('day', now())
group by 1
order by 1;

Calculate average bytes per current Raw row from the hypertable size, then project 30 × average registered rows/day plus 7 × average short-lifecycle rows/day. Add the existing non-Raw database size. The gate is:

projected database bytes <= 70% of the current PVC capacity

If the projection exceeds the gate, stop the rollout. Expand the PVC in a separate infrastructure change to the smallest 5-GiB increment that brings the projection to or below 70%. Do not rewrite an immutable StatefulSet volume-claim template merely because the bound claim was expanded. Verify a recoverable database backup or volume snapshot before continuing; a volume is not a backup.

Phase 1: stop ingress and drain

Change the environment’s GitOps values so that MQTT ingestion is disabled while Raw Storage and the Flow worker continue running. Synchronize the Application. Do not scale a GitOps-managed Deployment manually except under the established incident procedure.

Verify workload state without printing Secrets:

SENS_NAMESPACE=sens-test
sudo kubectl -n "$SENS_NAMESPACE" get deployment,statefulset,job,cronjob
sudo kubectl -n "$SENS_NAMESPACE" get pods

In the Ops portal or metrics backend, wait until both Raw Ingestion and Raw Stored report zero for nats_consumer_num_pending and nats_consumer_num_ack_pending. Confirm that sens_raw_storage_outbox_failures_total is no longer increasing and that the Raw Stored consumer acknowledgement floor reaches its delivered sequence.

Phase 2: run the guarded migration

Commit and synchronize this temporary GitOps state:

services:
  mqttIngestion:
    enabled: false
  rawStorage:
    enabled: false
  flowWorker:
    enabled: false

rawIngestionBootstrap:
  enabled: true
  migrateRawStoredRetention: true

rawIngestionRetention:
  enabled: true
  env:
    SENS_RAW_RETENTION_DAYS: "30"
    SENS_RAW_UNREGISTERED_DEVICE_RETENTION_HOURS: "168"

The bootstrap-value hash creates a distinct Job. Inspect the newest Job and its bounded log output:

sudo kubectl -n "$SENS_NAMESPACE" get jobs \
  -l app.kubernetes.io/component=raw-ingestion-bootstrap \
  --sort-by=.metadata.creationTimestamp
sudo kubectl -n "$SENS_NAMESPACE" logs job/<exact-bootstrap-job-name>

Success reports rawStoredRetentionMigration: "migrated" or, on a safe rerun, "already_migrated". Any other result blocks restart. Do not manually delete the stream to bypass a failed guard.

Phase 3: drain the database outbox

Set migrateRawStoredRetention=false, enable Raw Storage and the Flow worker, keep MQTT ingestion disabled, and synchronize again. Confirm:

• both workers become Ready;
• pending Raw Stored outbox rows drain;
• sens_raw_storage_outbox_published_total increases as expected;
• sens_raw_storage_outbox_failures_total does not increase;
• no raw-stored outbox publication failed log recurs;
• acknowledged Raw Stored references make nats_stream_total_messages{stream_name="SENS_RAW_STORED_V1"} return to zero;
• an intentionally unacknowledged test reference remains present until it is acknowledged.

Useful Kubernetes checks are:

sudo kubectl -n "$SENS_NAMESPACE" rollout status \
  deployment/sens-platform-test-raw-storage-worker
sudo kubectl -n "$SENS_NAMESPACE" rollout status \
  deployment/sens-platform-test-flow-worker
sudo kubectl -n "$SENS_NAMESPACE" logs \
  deployment/sens-platform-test-raw-storage-worker --since=30m

Use the corresponding release-specific Deployment names in Production.

Phase 4: resume and observe

Enable MQTT ingestion through GitOps and synchronize. Verify its readiness and then observe Test for at least 24 hours:

sudo kubectl -n "$SENS_NAMESPACE" rollout status \
  deployment/sens-platform-test-mqtt-ingestion-worker

Monitor stream message count, consumer pending and acknowledgement pending, Raw Stored outbox publications and failures, retention-CronJob results, Raw hypertable size, and PVC free capacity. Only after Test passes this gate may the same four phases be repeated for Production.

Recovery and rollback

• Before stream deletion, every failure is non-destructive and leaves the legacy stream intact.
• If recreation fails after deletion, rerun the same bootstrap image. A missing stream is a safe migration no-op, and normal bootstrap recreates the WorkQueue and consumer.
• PostgreSQL outbox rows are not deleted by this operation. Raw Storage republishes unpublished rows after restart.
• Do not deploy an older bootstrap against the WorkQueue; its immutable retention expectation is incompatible. Recovery is a forward fix.
• Returning to limits retention requires another complete drain and guarded recreation procedure.
• Do not lower SQL retention as an emergency rollback. Suspend the CronJob and forward-fix instead, because the next run could irreversibly delete rows.
• Do not run the policy migration down in Production. The Raw CronJob remains the authoritative lifecycle mechanism.
• Never delete the NATS PVC, TimescaleDB PVC, raw rows, or idempotency keys as part of this stream migration.