Observability

A production service needs security, configurability, observability, recoverability and other operational controls in addition to features. This page describes the implemented telemetry, known gaps, and design trade-offs. The separate verification guide maps telemetry to quality scenarios and records dashboard evidence.

Coverage in one glance:

1. Health Check API

Problem. Process liveness is not the same as readiness to serve traffic. Restarting a healthy process because Postgres is briefly unavailable creates a restart loop, while routing requests to an instance that has not finished wiring its adapters creates avoidable failures.

Implementation. Every service exposes three endpoints instead of overloading one /health. GET /health is a static liveness answer, GET /ready runs the dependency checks that the service currently implements and returns 503 with a per-check breakdown, and GET /metrics is the Prometheus exposition. Endpoint presence does not imply complete dependency readiness.

Readiness is not uniform, because the dependencies are not. passport-service, fridge-service, shopping-list-service and measurement-pipeline probe Postgres and Kafka; the three TypeScript services probe Postgres only; api-gateway holds no database or broker connection of its own, so its /ready is the static handler. Two gaps follow: the TypeScript services use Kafka without probing it, and their checks have no timeout, where the Go implementation bounds each one at 2 seconds (packages/observability/go/readiness.go).

External food and recipe APIs are excluded from readiness by policy. A scan degrades gracefully without them (QAS-2), so an outage there must never drain a service that can still serve passports. Compose declares container healthchecks against /health only; Kubernetes declares a livenessProbe on /health and a readinessProbe on /ready, which is where draining actually happens.

Evidence. packages/observability/go is unit tested, Identity tests its local TypeScript observability.ts, and scripts/journey-test.sh asserts all three endpoints on running containers. Identity's test is not evidence that the other two duplicated TypeScript modules remain identical; each copy needs coverage or a drift check.

Trade-off. Dependency checks add load and can hang, so they are bounded and cover only what the service contract needs. A synthetic transaction would catch more, at the cost of running external-provider work every probe interval.

2. Application Metrics

Problem. Health endpoints answer only the present binary state. Quality scenarios and service levels need rates, latency distributions, failures and saturation over time, without recording personal data.

Implementation. Collection uses the pull model: services expose /metrics and Prometheus scrapes all eight every 15 seconds (deployment/prometheus/prometheus.yml), rather than pushing to a metrics service. Prometheus evaluates the SLI recording rules in deployment/prometheus/rules/slo.yml, and Grafana provisions its datasource and dashboard from disk.

Instrumentation sits only at the adapter and middleware seam, never in a service core. In Go it comes from the shared packages/observability/go; in TypeScript from a small per-service observability.ts (Fastify hooks plus prom-client), duplicated rather than shared because those images are self-contained (see section 7). Passport times each passport section from internal/adapters/outbound/sectionmetrics, a decorator around the VerdictGateway, JourneySource and RecipeSource ports, which keeps the core free of the metrics dependency and times every concrete source uniformly.

The metrics. Three are fleet-wide, emitted by every instrumented service: http_requests_total{service,method,route,status_class}, which also carries SLI-2, http_request_duration_seconds{service,method,route} and http_in_flight_requests{service}.

Passport owns the scan path: packytrace_scan_duration_seconds{outcome}, whose explicit 3s bucket is the source of SLI-1 and QAS-1; packytrace_scan_requests_total{outcome}; packytrace_passport_section_duration_seconds{section,outcome} for section-level diagnosis; and, for QAS-2, packytrace_external_source_requests_total{source,outcome}, packytrace_external_source_duration_seconds{source,outcome} and packytrace_circuit_breaker_open{source}.

Personalization owns the verdict and erasure metrics: packytrace_verdict_requests_total{outcome} and packytrace_verdict_computation_duration_seconds{outcome} for QAS-1, and erasure_propagation_total{outcome} for QAS-3.

Each service additionally exposes its client library's default collectors ('go_', 'nodejs_', 'process_'); no dashboard panel depends on them. One caveat: http_request_duration_seconds* uses different default buckets in the two fleets, so a quantile computed across both at once is approximate. Neither SLI is affected, because SLI-1 reads the passport histogram with its own explicit buckets and SLI-2 reads a counter.

Label discipline (privacy wall). Only bounded labels are permitted: service, method, route template, status_class, outcome, source, section. Never visitorId, accountId, scanId, gtin, raw Digital Link, token, email, health-profile fields, or brandId. Route templates keep cardinality finite, and the privacy rule keeps per-scan and per-visitor facts out of telemetry.

Trade-off. Bounded labels protect privacy and cardinality but prevent per-user debugging. That loss is intentional under the privacy wall.

3. Log Aggregation

Problem. The evidence for one user-visible failure is split across the gateway log and the log of whichever service owned the request, and a container's log disappears with the container.

Built: the developer half. Every service logs structured records to stdout and writes no files, so the platform decides where output goes. Go services build a log/slog JSON handler bound to service; the TypeScript services use Fastify's built-in pino logger, also JSON. Those streams are read with docker compose logs --follow locally and on the AWS box, and kubectl logs on the kind cluster.

Not built: the operations half. There is no pipeline and no log server: no Loki, Promtail, ELK or CloudWatch anywhere in deployment/, and no log-level configuration. One detail would have to be fixed first: each Go router also mounts chi's middleware.Logger, which writes line-oriented text into the same stdout stream as the slog JSON, so a Go container emits two formats.

Trade-off and first step. Direct Compose logs may be sufficient for a demonstration, but they are not durable production log aggregation. First emit one structured format, then choose a retained log backend and access policy based on recovery and privacy requirements.

4. Distributed Tracing

Problem. When a scan is slow, the latency has to be attributed to a hop. A profiler cannot see across processes, so the attribution has to travel with the request.

Not built. There is no OpenTelemetry, Zipkin or Jaeger instrumentation and no tracing server in any deployment target.

What exists. Every inbound request gets an id, from chi's middleware.RequestID in Go and Fastify's request.id in TypeScript, returned to the client in error bodies as correlationId and declared in every service openapi.yaml. It ties a user report to a log line in one service, and stops there: the gateway's httputil.ReverseProxy does not inject the id downstream and no outbound adapter forwards it, so passport's id for a scan is unrelated to the gateway's. The asynchronous side has the same gap by another route: publishers currently set the envelope correlationId to a domain identifier (scanId, itemId, or accountId) and never set causationId. Domain/person identifiers must not be copied into telemetry. Replace this with an opaque request/correlation value and define propagation, retention and redaction before adding a trace backend.

Trade-off. The deepest synchronous path is gateway, then passport, then at most one downstream call. packytrace_passport_section_duration_seconds{section} already answers where time was spent by section, using bounded metric labels. Trace attributes require their own cardinality and privacy rules; tracing does not remove that obligation. The pattern would cost two instrumentation stacks plus a collector and trace store on a single-instance deployment.

First step. Propagate one X-Request-Id through the proxy, the outbound adapters, the log records and the envelope, and set causationId when an event reacts to another. That is the precondition for OpenTelemetry; a tracing library added before it would produce disconnected spans.

5. Exception Tracking

Problem. An exception is either a bug or the symptom of a failure, and both need to reach a human with de-duplication and a resolution state. Logs are a poor medium: stack traces do not fit line-oriented search, and one repeated failure looks like thousands of entries.

Not built. There is no exception tracking service, and no alerting of any kind: deployment/prometheus/rules/slo.yml holds recording rules and no alerting rules, neither Compose file runs an Alertmanager, and the dashboard declares no alert rules. Someone has to look at the dashboard.

What exists. Panics are contained rather than fatal, by chi's middleware.Recoverer and Fastify's error handler, which turn a panic or unhandled rejection into a 500 and a logged error while the process keeps serving. Error responses carry a stable code and the correlationId from section 4. One failure class was promoted from a log line to a metric, erasure_propagation_total{outcome}, because nobody would otherwise see it.

Trade-off and first step. A dashboard without a watcher is not notification. Define an owner and response channel, then add Alertmanager burn-rate alerts for the recorded SLIs. Exception tracking can be evaluated separately for code-level failures.

6. Audit Logging

Problem. Some actions must be reconstructable afterwards: who consented to what, when, and under which policy version. A mutable table answers "what is true now" and destroys the answer to "what happened".

Implementation. Of the three usual implementations (audit calls in business logic, aspect-oriented interception, event sourcing) PackyTrace takes the third where history is part of the domain, and adds a database-enforced ledger for the compliance-facing case:

  • fridge_events and shopping_list_events are append-only logs of what an account did, with the item tables as projections written in the same transaction. The audit trail is a by-product of the storage decision, not a parallel mechanism. See Event Sourcing.
  • consent_ledger in identity-service records every grant and revocation with kind, granted, method, policy_version and created_at, and a BEFORE UPDATE OR DELETE trigger raises 'consent_ledger is append-only'. It is the only audit guarantee here that does not depend on application code behaving.

scan_records is mutable operational data, and the Kafka topics run with default retention, so neither is a store of record.

Trade-off. There is no fleet-wide “who did what” log and no actor recorded for administrative actions. The privacy wall limits what may enter telemetry, but it does not remove the need for a privacy-reviewed security audit trail. Define auditable operations, authorized readers, retention and redaction before calling this production-ready.

7. Shared observability infrastructure

Problem. Health endpoints, metrics middleware and readiness probes are identical in every service. Reimplementing them per service invites drift; putting them in the service core would break the dependency rule.

Implementation. packages/observability/go is a workspace module wired through go.work, exporting Middleware, Handler, MountReady, ReadyHandler and the Check type. It is infrastructure rather than a bounded context, and a '-core-purity' depguard rule keeps service cores from importing it. It does not cover the startup path: the /health handler, logger construction, configuration and graceful shutdown stay in each cmd/server/main.go, because pulling them in would move startup policy out of the service that owns it. The TypeScript side carries a local observability.ts* implementation in each service.

The Go package conflicts with the categorical rule that services share only contracts. Until the maintainer resolves that policy, this paragraph describes the repository but does not authorize additional shared runtime packages.

Service mesh. The repository does not use one. A mesh could centralize transport telemetry, mutual TLS, and retry policy, but it would not replace domain metrics or application-specific readiness. Reassess it if the deployment topology or transport requirements change.

8. Verification and dashboard access

The mapping from metrics to QAS scenarios, screenshot provenance and dashboard access instructions are maintained separately in Observability Verification. This page remains the design and implementation-gap reference.