Troubleshooting

Start with the user-visible symptom and move one dependency boundary at a time. Do not print Secrets, raw credentials, access tokens, or unnecessary customer payloads while investigating.

First five facts

Record these before changing anything:

1. environment and public hostname;
2. UTC time range;
3. affected tenant or organization ID, when safe;
4. correlation ID or raw event ID;
5. deployed version from /version and Argo CD revision.

Symptom routing

Symptom	Check in order	Detailed reference
Docs or web console is unreachable	DNS/TLS, Traefik Ingress, Service endpoints, Pod readiness.	Kubernetes Deployment
Browser login fails but API health is green	Browser Origin/CORS, login response, auth metrics, database-backed user/session state.	Authentication
API returns 401	Access-token presence/expiry, session and user status.	API Authentication
API returns 403	Current role, tenant, organization grant, and requested scope.	Tenancy model
API returns 503 on an administrative write	Database readiness and fail-closed audit persistence.	platform-api
Secret changed but Pod uses the old value	ExternalSecret readiness, target Secret metadata, Reloader event, rollout.	Secret Management
Argo Application is OutOfSync	Desired revision, diff, sync policy, and whether production is intentionally pending.	Argo CD Applications
Migration blocks a release	Migration Job logs, migration history, SQL compatibility, forward-fix plan.	Database Migrations
No new raw messages arrive	MQTT readiness, subscription metric, excluded customer, JetStream publish signals.	MQTT worker
JetStream backlog grows	Raw worker readiness, database latency/errors, pending and ack-pending gauges.	Raw-storage worker
Duplicate MQTT delivery appears	Expected at QoS 1; verify a single idempotent raw row and duplicate metric.	Netmore Raw Ingestion
Unknown-device raw messages grow	DevEUI in raw metadata versus active device registry entry.	Data model
API console cannot send a request	Selected server, browser mixed-content/CORS error, exact docs Origin allowlist.	Documentation service

Safe status checks

Use status endpoints and object conditions before logs:

curl -fsS https://api.dev.iot-sens.schlossers.at/healthz
curl -fsS https://api.dev.iot-sens.schlossers.at/readyz
curl -fsS https://api.dev.iot-sens.schlossers.at/version
sudo kubectl -n sens-test get deploy,statefulset,pod
sudo kubectl -n sens-test get secretstore,externalsecret

Then inspect only the owning service with the time range and correlation ID. Redact payloads and identifiers before sharing output outside the approved support context.

Recovery discipline

• Prefer read-only refresh and status checks first.
• Do not delete an Argo CD Application to clear a degraded card.
• Do not delete TimescaleDB or NATS PVCs during workload rollback.
• Do not edit generated Kubernetes Secrets; change the catalogued value through the Ops portal.
• Disable the MQTT receiver before deliberately interrupting the raw consumer or stream.
• Treat production sync, secret changes, and retention changes as production operations with review and monitoring.

Observability adapters

If Sentry or PostHog is unavailable, first verify that the corresponding enable flag and complete runtime configuration are present. The application must stay ready and functional; the Ops source becomes stale or unknown. Do not enable verbose SDK logging in Production or paste provider responses into tickets.

For missing central logs, search the application stdout locally by Correlation ID, then inspect Fluent Bit buffer and output counters. Absence in VictoriaLogs is not proof that no event occurred because V1 logs are not backed up. For SLO evidence, inspect the explicit evidence state before treating low coverage as a failure. ramp_up means sampling has met the threshold since its first point; incomplete means actual gaps; not_observed means a successful query without samples in the period; and unavailable means the evidence source could not be read. Verify the eu-primary series and preserve the Ops victoria-metrics-slo-data volume during recovery.

The public page offers 24-hour, seven-day, and 13-month history. If only the short-term views fail, inspect the allowlisted MetricsQL range queries for evaluable samples, available samples, primary coverage, and the first sample in each interval. Never infer an online interval from a missing bucket.