Event Sourcing¶

This page documents the repository-specific use of event sourcing in Fridge and Shopping List, the boundary between stored events and Kafka integration events, and the event-driven Measurement Pipeline. Canonical published payload structures remain in contracts/jsonschema.

Domain Events and Integration Events¶

The word "event" carries two different meanings in this system, and separating them is the first thing worth doing.

A domain event records a fact recognized by a bounded context. Fridge and Shopping List persist selected domain events as their source of truth; other contexts can use domain events without event-sourced persistence. Their current-item tables are projections built from the stored stream.

An integration event is a message on a Kafka topic announcing that something happened, so other services can react without the publisher waiting for them.

The two are related but are never the same object. A domain event is private to its service and shaped so it can be replayed. An integration event is public, versioned and small enough for the documented consumer contract. The separation prevents private replay shape from becoming a cross-service interface.

The Event Store as Source of Truth¶

A Fridge item is represented by its lifecycle facts: it is added and may later be consumed or discarded. A Shopping List item is added, may change quantity, and ends as either bought or removed.

If the services only stored the latest row, that story would be overwritten every time something happened. The waste summary could not say what was thrown away last month, and a future feature that needs historical state would have nothing to replay. A separate audit table would also require an additional consistency mechanism.

The append-only tables fridge_events and shopping_list_events are the event store, and they are the source of truth during normal aggregate operation. Existing facts are not updated by ordinary commands. The accepted AccountDeleted requirement must erase the affected account's state; the exact deletion mechanism is pending the retention decision recorded in the audit. The tables fridge_items and shopping_list_items are the projection: they hold the current state and are treated as disposable, because they can always be recomputed from the store.

The Write Model: Command, Aggregate, Event¶

Diagram source: docs/development/architecture/event-sourcing.drawio. Current implementation view, verified 2026-08-01. The post-commit publication gap is shown explicitly; make docs-diagrams regenerates the image.

Every command that changes something, whether it is adding an item, consuming it, discarding it, changing a quantity, marking it bought or removing it, goes through the same six steps.

1. The request arrives through the gateway and reaches the use case with the account already identified.
2. The use case calls Rehydrate(accountId), which reads every event for that account in the order the database assigned.
3. Replay(events) folds that list into the current state of the aggregate. It is a plain function over a slice of events: it knows nothing about Postgres, so it can be tested on its own.
4. The aggregate checks the invariant against the replayed state. For example, an item cannot be consumed twice, and the same serial cannot sit in the fridge twice.
5. If the invariant holds, the aggregate returns the new state together with the domain event it decided to record, a PendingEvent carrying the type, the item, the payload and the timestamp. The repository appends that event verbatim and updates the projection row inside one Postgres transaction.
6. Once that transaction commits, the service publishes the integration event to Kafka.

In step 5, the type and payload of the history are chosen by the core, not by the persistence adapter. The repository never decides what happened, it only stores what it was handed, which is what keeps the log a record of domain decisions rather than a by-product of saving rows.

If the invariant does not hold, nothing is written.

Aggregate Rehydration: Deciding on Replayed State¶

The aggregate object does not survive between requests; its ordered facts do. PackyTrace uses two terms for reconstruction:

Rehydrate(accountId) = load the ordered event stream + Replay(events)

Replay is the deterministic fold; Rehydrate includes loading the ordered stream. Commands decide against rehydrated state, not the disposable projection. Reads such as active items and freshness use the projection; the monthly waste summary reads terminal events because the projection holds only current status.

SQL predicates and the Fridge duplicate-serial partial index are the current concurrency backstop. Two commands can replay the same unversioned stream, but only one matching projection update succeeds; the other receives ErrItemNotFound or ErrAlreadyInFridge. Expected-version checking on the event append is not implemented and would require a separate decision.

Atomicity: One Transaction for Event and Projection¶

The event append and the projection update happen in the same database transaction, so either both land or neither does. There is no window in which the store says an item was consumed while the projection still shows it as bought. The projection is strongly consistent with the store. There is no asynchronous projector or snapshot mechanism; full streams are replayed at the current scale.

Event Payload Design: Stored Event versus Published Event¶

ItemAddedToFridge has two representations. The private stored event includes the product snapshot needed to rebuild state without calling Passport. The published integration event contains only the versioned fields defined in contracts/jsonschema. Core types (storedItemAdded, storedItemStatus) shape stored history; generated contract types are used at the Kafka adapter.

Projection Rebuild and Replay Determinism¶

Each repository exposes RebuildProjection(accountId). In one transaction it loads the stream, deletes that account's projection rows, performs the deterministic replay and inserts the result. Core tests prove the folds. Repository tests selected by TEST_DATABASE_URL rebuild against a disposable Postgres and compare the result.

Trade-offs of Event Sourcing¶

History and current state are stored twice, and old payloads must remain replayable after schema changes. Only Fridge and Shopping List use event sourcing. Their item lifecycles are stored as streams; Fridge alerts use the ordinary fridge_alerts table because they are notification state, not aggregate history. The other bounded-context services use current-state tables.

Published Language: Versioned Event Contracts¶

Once a write has committed, the service announces selected facts on its own Kafka topic. Services use internal REST for synchronous queries or commands and Kafka for asynchronous facts; they never read each other's tables. The publication happens after the commit and outside the transaction, with no outbox table, so a broker outage can lose the announcement while the fact remains in an event-sourced producer's store. There is no automated republisher today. A manual replay can reconstruct Fridge or Shopping List publications, but Passport, Personalization and Identity facts cannot be recovered this way because those services are not event-sourced. A transactional outbox or equivalent durable publication mechanism remains required hardening. The only thing a producer and a consumer share is the event's versioned JSON Schema, which makes those schemas the published language of the whole system.

Every message uses one shared envelope, envelope.schema.json, wrapped around one payload. The v1 payloads in contracts/jsonschema/events/v1/, grouped by what they are about:

• Scanning and verdicts: ProductScanned, VerdictComputed.
• Fridge: ItemAddedToFridge, ItemConsumed, ItemDiscarded, ItemExpiring, ItemExpired, AlertRaised.
• Shopping list: ItemAddedToShoppingList, ShoppingListItemBought, ShoppingListItemRemoved.
• Identity and consent: ConsentGranted, ConsentRevoked, AccountDeleted, VisitorLinkedToAccount.
• Health profile: HealthProfileUpdated, HealthProfileDeleted.
• Brand analytics: BrandMetricBatchPublished, the only event allowed to cross the privacy wall, because it carries minimum-group-size aggregates instead of individual facts.

Each service publishes onto its own topic: passport.facts.v1, identity.facts.v1, personalization.facts.v1, fridge.facts.v1 and shopping-list.facts.v1. The measurement pipeline publishes its gated aggregates onto measurement.brand-metrics.v1.

Current consumption is limited. personalization-service listens to identity.facts.v1 so consent changes reach the health profile, and the measurement pipeline listens to the scan and verdict topics. Fridge, Shopping List and Identity facts are published and available, but nothing projects them into brand metrics today.

The schemas live in the repository rather than on this site. contracts/README.md holds the rules, the root CONTRACTS.md follows one event from producer to consumer, and contracts/gen/ holds the generated Go and TypeScript types. Those generated files are never edited by hand: make contracts-gen rewrites them and CI fails if they have drifted.

Schema Evolution and Backward Compatibility¶

1. Payloads grow additively. New fields are optional, existing fields never change meaning, and nothing is removed.
2. A breaking change becomes a new version under events/v2/ and, when necessary, a versioned topic. Producers dual-publish or provide an explicit migration path until every consumer has moved. Retirement requires a recorded consumer inventory, retention period and rollback plan.
3. Consumers tolerate unknown fields, which is what makes rule 1 safe for producers.
4. Payloads carry envelope data and domain reason codes only. Never entities, never database models, never text meant to be shown to a person.

Event-Driven Architecture: the Measurement Pipeline¶

The pipeline is the service built entirely around events. It consumes facts, aggregates them over time windows, and publishes only aggregates that are large enough to be anonymous. Passport and Personalization keep serving their normal request paths and publish facts as a side effect, so producer and consumer stay temporally decoupled.

A fact moves through the pipeline in seven stages.

1. Consume. A franz-go consumer group subscribes to the configured input topics, and the inbound adapter decodes the shared envelope into the core's Fact type.
2. Project. A projector registry maps ProductScanned to scans_per_product and VerdictComputed to grade and goal metrics. Anything else produces no increments.
3. Deduplicate. This is the idempotent consumer. The event ID is inserted into processed_event with ON CONFLICT DO NOTHING, in the same transaction as the counter increments, which is what makes Kafka redelivery harmless.
4. Aggregate. Increments upsert durable Postgres counters keyed by time window, brand, metric and canonical dimensions. The default is a weekly tumbling window.
5. Gate. The flush use case picks up only closed groups that reached MIN_GROUP_SIZE, five by default, and the core checks that threshold again before anything leaves. Groups below it are finalized and never emitted. This is the privacy wall in code.
6. Publish. Eligible groups become BrandMetricBatchPublished events, and a group is marked emitted only after the broker acknowledges the publication.
7. Build the read side. Brand Analytics consumes that topic and upserts its own tenant-scoped read model, which is what the Brand dashboard queries.

The pipeline is also required to reject CommunityBrandID before publication. That filter and its negative privacy-boundary test are not implemented yet, so community-sourced scans must not be presented as verified Brand analytics.

The Measurement consumer commits offsets only after successful processing, so Kafka redelivery is possible and the stage-3 idempotency ledger prevents double counting. A failed batch publication leaves the group eligible for the next flush. Brand Analytics currently logs a failed ingest without an application-level retry; replaying from the earliest retained offset with a new consumer group can rebuild the read model. Retry and dead-letter handling remain explicit gaps.

Kafka keeps analytics outside the synchronous scan response. The broker separates the Consumer journey from slower aggregation, retains integration facts for later consumers, and gives the privacy wall a place to transform facts before the brand side ever sees them. Local and Kubernetes deployments run Apache Kafka; the smaller AWS Compose deployment runs Redpanda, speaking the same protocol with the same topics and contracts.

What Brand Analytics does with that aggregate topic, and why its read model is allowed to lag behind the consumer side, is the CQRS pattern described in Microservices and Patterns.