Runbook

What to check when PackyTrace misbehaves, in any environment. Deployment mechanics live in AWS Application and Kubernetes (kind); this page is only the "it is broken, now what" path.

1. First look

Environment Status Logs
Local compose make health make logs
Kubernetes (kind) make k8s-status make k8s-logs
AWS demonstration make aws-status SSM onto the box, then docker compose logs

Every service exposes three endpoints (Observability):

  • /health: liveness. Static "ok"; if this fails, the process is down.
  • /ready: readiness. Returns 503 when one of the dependencies checked by that service is unavailable. Go stateful services check Postgres and Kafka; TypeScript services currently check Postgres only; the gateway's handler is static. External food/recipe APIs are excluded by the readiness policy; their failure degrades passport sections, not the service.
  • /metrics: Prometheus exposition, scraped every 15s.

2. Dashboards

  • Local: Grafana at http://localhost:3000 (anonymous viewing on), Prometheus at http://localhost:9090 (Status, then Targets, to confirm scrapes).
  • AWS: 'grafana.' (admin password in SSM) and 'prometheus.' (Caddy basic auth). See AWS Application for the password commands.

Grafana opens on PackyTrace SLOs & QAS. Read it top-down: the SLI-1/SLI-2 compliance row first (SLOs & SLIs), then the panel that explains the miss: scan latency p95, per-section passport latency, external source outcomes, circuit-breaker state, fleet HTTP rate by status class.

Metrics only appear once traffic flows. On a fresh stack, drive a few scans through the gateway and wait one scrape interval.

3. Known failure modes

Every brand QR resolves as "unknown product". Brand catalogs are never stored in the database; passport-service syncs them into memory from the producer simulators because each producer remains authoritative for its own products. If the simulators are down, or the service restarted while they were unreachable, brand products stay unresolved until one sync succeeds. Check the four mock-brand-service containers and BRAND_PROVIDER_URLS, then wait one configured synchronization interval. Restart passport-service only if periodic synchronization does not recover. Scans, identity, and community (Open Food Facts) products keep working throughout.

A service crash-loops on startup with a migration error and the gateway returns 502. Identify the exact service and migration from its logs. Do not delete rows manually. Fix forward with an idempotent migration or restore from a verified backup. The former duplicate catalog-seed incident predates the current producer-catalog design and is not a general recovery procedure.

/ready returns 503 on several services. Read each response's named checks. If the same Postgres or broker check fails across stateful services, diagnose that shared dependency; do not assume every service probes both.

The brand dashboard is empty. Aggregates only cross the privacy wall when a window closes with at least MIN_GROUP_SIZE (default 5) events, and the flush runs on an interval. Locally, make seed publishes demo facts through the Measurement Pipeline; it is not repeatable against a running stack (published windows are never re-emitted). To re-seed, make up-build (wipes local volumes) and seed again.

Scan latency SLO burning. Check the per-section passport latency panel: if one section (verdict, journey, recipes) is slow, the culprit is that section's source, and the external-source and circuit-breaker panels tell you whether the breaker already opened. The scan path is the HPA target on Kubernetes; check make k8s-status for scaling state.

4. Reset levers, mildest first

  1. Restart the one unhealthy workload after replacing '' with its exact Compose or Kubernetes name: 'docker compose restart ' or 'kubectl --context kind-packytrace -n packytrace rollout restart deployment/'.
  2. make down then make up: keeps data volumes.
  3. make up-build: full local reset, wipes the local database.
  4. Kubernetes: make k8s-down then make k8s-up: recreates the cluster from scratch.
  5. AWS: make aws-stop / make aws-start preserves attached storage. terraform apply may update in place or replace resources depending on the plan; inspect and approve the plan before applying it.