The Transactional Outbox Pattern: Solving the Dual-Write Problem

How to publish business events reliably without distributed transactions, and why you still need to handle duplicates.

Imagine an order service that receives an API request, saves a new order in PostgreSQL, and publishes an OrderPlaced event to Kafka. Other services use that event to reserve inventory, initiate payment, or update a customer-facing timeline.

One day, the database commit succeeds, but the application crashes before publishing the event. The API may even have returned an error, leaving everyone unsure whether the operation happened. The order exists, but the rest of the system never hears about it.

This isn’t an unusual edge case. It’s a fundamental problem whenever one business operation needs to change a database and send a message to another system.

The Transactional Outbox pattern solves the reliability gap by changing what we commit: instead of trying to atomically write to the database and broker, we atomically write business data and a durable record of the event to be published into the same database. A separate process delivers the event later.

That simple idea is powerful. The details, however, matter.

The dual-write problem

A straightforward implementation looks like this:

@Transactional
public Order placeOrder(CreateOrder command) {
    Order order = orderRepository.save(new Order(command));
    
    kafkaTemplate.send("orders", order.id().toString(),
                       new OrderPlaced(order.id()));
    
    return order;
}

@Transactional
public Order placeOrder(CreateOrder command) {
    Order order = orderRepository.save(new Order(command));
    
    kafkaTemplate.send("orders", order.id().toString(),
                       new OrderPlaced(order.id()));
    
    return order;
}

The @Transactional annotation only manages the database transaction in this typical setup. It does not magically create one atomic transaction spanning PostgreSQL and Kafka.

If we save first and publish second, the database can commit and the publish can fail.

Publishing before committing does not solve it: the message can be accepted and the database transaction can subsequently roll back.

Even waiting for Kafka’s acknowledgement does not eliminate the gap. Acknowledgement tells us something about the broker write, not whether the database and broker committed as a single unit.

A distributed transaction using two-phase commit (2PC) is a theoretical approach when all participants support the necessary protocol. In practice, it adds coupling and operational complexity, and it is not generally available across arbitrary databases and brokers. The outbox pattern deliberately avoids requiring 2PC.

The key idea: make the intention to publish durable

The application performs two writes inside one local database transaction:

Update the business entity, such as an order.
Insert an event into an outbox_events table.

Either both changes commit or neither does. A separate relay reads committed outbox entries and sends them to the message broker.

This establishes the crucial invariant:

If the business transaction commits, the corresponding event is durably available for publication. If the transaction rolls back, that event is not available for publication.

Notice the wording. The database and broker are not updated atomically. Instead, after a successful commit, we have a durable obligation to publish. Delivery might be delayed while the broker or relay is unavailable; the guarantee depends on retaining the outbox record, eventually recovering the delivery infrastructure, and operating the relay correctly.

This also means consumers are eventually consistent with the producer. The order can be visible in the database before its event appears in Kafka.

A concrete example using PostgreSQL and Spring Boot

Start with an outbox table. This is one possible schema, not a universal standard:

CREATE TABLE outbox_events (
    id                UUID PRIMARY KEY,
    aggregate_type    TEXT NOT NULL,
    aggregate_id      TEXT NOT NULL,
    event_type        TEXT NOT NULL,
    schema_version    INTEGER NOT NULL DEFAULT 1,
    payload           JSONB NOT NULL,
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at      TIMESTAMPTZ
);

CREATE INDEX idx_outbox_unpublished
    ON outbox_events (created_at, id)
    WHERE published_at IS NULL;

CREATE TABLE outbox_events (
    id                UUID PRIMARY KEY,
    aggregate_type    TEXT NOT NULL,
    aggregate_id      TEXT NOT NULL,
    event_type        TEXT NOT NULL,
    schema_version    INTEGER NOT NULL DEFAULT 1,
    payload           JSONB NOT NULL,
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at      TIMESTAMPTZ
);

CREATE INDEX idx_outbox_unpublished
    ON outbox_events (created_at, id)
    WHERE published_at IS NULL;

The id identifies an individual event and is essential for deduplication. The aggregate_id represents the business entity (such as an order) and is a natural Kafka key when per-entity ordering matters. aggregate_version can help consumers detect stale or out-of-order updates. The schema_version lets producers evolve their event contracts explicitly.

The service writes the order and event together:

@Service
class OrderService {
    private final OrderRepository orders;
    private final JdbcTemplate jdbc;
    private final ObjectMapper mapper;

    // Constructor injection omitted for brevity.

    @Transactional(rollbackFor = Exception.class)
    public UUID placeOrder(CreateOrder command) throws JsonProcessingException {
        Order order = orders.save(Order.from(command));
        UUID eventId = UUID.randomUUID();

        String payload = mapper.writeValueAsString(
            new OrderPlaced(order.getId(), order.getTotal())
        );

        jdbc.update("""
            INSERT INTO outbox_events
              (id, aggregate_type, aggregate_id, event_type, payload)
            VALUES (?, 'Order', ?, 'OrderPlaced', ?::jsonb)
            """, eventId, order.getId().toString(), payload);

        return order.getId();
    }
}

@Service
class OrderService {
    private final OrderRepository orders;
    private final JdbcTemplate jdbc;
    private final ObjectMapper mapper;

    // Constructor injection omitted for brevity.

    @Transactional(rollbackFor = Exception.class)
    public UUID placeOrder(CreateOrder command) throws JsonProcessingException {
        Order order = orders.save(Order.from(command));
        UUID eventId = UUID.randomUUID();

        String payload = mapper.writeValueAsString(
            new OrderPlaced(order.getId(), order.getTotal())
        );

        jdbc.update("""
            INSERT INTO outbox_events
              (id, aggregate_type, aggregate_id, event_type, payload)
            VALUES (?, 'Order', ?, 'OrderPlaced', ?::jsonb)
            """, eventId, order.getId().toString(), payload);

        return order.getId();
    }
}

This assumes the OrderRepository and JdbcTemplate participate in the same PostgreSQL transaction and transaction manager.

The method’s successful return is not the commit itself; with Spring’s normal transaction interception, commit happens when the transactional method invocation completes. If the outbox insert fails, the order change must roll back too. The rollbackFor setting is deliberate: by default, Spring does not roll back for checked exceptions such as JsonProcessingException.

There are many ways to implement the domain model. Implementing this pattern using a traditional Spring Data JPA Repository would be equally valid. The most important property is not the Java code: it is that the business write and outbox insert share one atomic commit.

Getting events from the outbox to the broker

There are two common approaches. They solve the same problem but have different operational characteristics.

Option A: Polling publisher

A background worker periodically looks for unpublished events, sends them to the broker, waits for a successful broker acknowledgement, then marks them as published.

Conceptually:

repeat:
    events = claimAvailableBatch()   // commit short DB transaction
    for event in events:
        try:
            publishToBrokerAndWaitForAcknowledgement(event)
            markPublishedIfStillOwner(event)
        catch transientFailure:
            releaseOrScheduleRetry(event)

In a simple single-worker prototype, published_at IS NULL can identify unpublished records.

With multiple workers, you need a concurrency-control strategy so they do not all pick the same event at once. PostgreSQL’s FOR UPDATE SKIP LOCKED is useful when claiming queue-like records:

SELECT id FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at, id
FOR UPDATE SKIP LOCKED
LIMIT 100;

SELECT id FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at, id
FOR UPDATE SKIP LOCKED
LIMIT 100;

That query alone is not a production-ready claim protocol. It must run inside a short transaction that atomically claims the rows, often using a status or lease owner and expiry. Commit the claim before publishing; do not hold database row locks while waiting on network I/O. Make the final acknowledgement conditional on still owning the claim. Expired claims must be recoverable, and retries may publish duplicates.

Strengths: straightforward to reason about, uses normal SQL, needs no CDC platform, can be appropriate for modest event rates.

Costs: repeated database queries, tuning batch sizes and intervals, cleanup of published rows, concurrent worker coordination, and potential publication-order complexity.

Option B: Change Data Capture (CDC)

Instead of querying the outbox table repeatedly, a CDC process reads committed changes from the database transaction log. With PostgreSQL, a tool such as Debezium can consume logical changes from the write-ahead log (WAL) and publish inserted outbox events to Kafka.

Debezium offers an Outbox Event Router transformation that maps an outbox table’s event ID, aggregate key, type and payload into broker records. Its default conventions use column names such as id, aggregatetype, aggregateid, type, and payload, so the example schema above would require custom mapping or adaptation.

Strengths: avoids polling the application table, follows committed database changes, can achieve low delivery latency at scale.

Costs: CDC infrastructure, connector and schema configuration, replication-slot monitoring, database-specific behavior, and operational complexity during failover or recovery. In PostgreSQL, an unhealthy replication slot can retain WAL and consume disk space.

For Debezium’s outbox router, treat the outbox primarily as an insert-only event stream: its documented transformation expects INSERT operations and filters DELETEs. That differs from the polling schema’s published_at update lifecycle. Choose the lifecycle that fits the relay; do not assume one implementation can be swapped for the other unchanged.

Consideration	Polling publisher	CDC / log tailing
Infrastructure	Application worker + database	CDC connector and log access
Latency	Influenced by polling interval	Often lower, driven by log streaming
Database load	Queries, claims and status updates	WAL/logical-decoding overhead
Portability	Broad SQL support	Database and connector specific
Recovery	Pending rows and worker state	Connector offsets, WAL retention
Complexity	Simpler to start; harder at high concurrency	More infrastructure to operate

Neither approach automatically provides end-to-end exactly-once business processing.

The unavoidable duplicate window

Suppose the relay sends event E-42 and Kafka acknowledges it. Immediately afterward, the relay crashes before updating the outbox record as published.

On restart, that record still looks unsent. The relay publishes E-42 again.

Both sends are reasonable from the relay’s point of view. The alternative, marking the record published before receiving broker acknowledgement, is worse, because an unsuccessful send can then be permanently lost.

So the usual delivery target is at-least-once publication, with consumers designed to tolerate duplicates.

A common consumer solution stores the processed event ID with the business operation in the same consumer-side transaction:

CREATE TABLE processed_events (
    consumer_name TEXT NOT NULL,
    event_id      UUID NOT NULL,
    processed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, event_id)
);

CREATE TABLE processed_events (
    consumer_name TEXT NOT NULL,
    event_id      UUID NOT NULL,
    processed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, event_id)
);

When processing event E-42, the consumer begins a transaction, inserts (consumer_name, E-42), performs its business update, and commits. If the unique-key insert indicates the event was already processed, it skips the business update.

This works when both writes share a transactional data store. If the handler calls an external payment API, the consumer cannot atomically commit its local deduplication row and that remote side effect. Use an idempotency key supported by the provider, or design an explicit recovery and reconciliation flow.

Exactly-once is a scope-dependent claim. Broker transactions may provide exactly-once processing within a defined broker workflow, but that is not the same as exactly-once side effects across independent databases and external APIs.

Ordering is a separate problem

Suppose an order changes from CREATED to PAID and then CANCELLED. If consumers see OrderCancelled before OrderPaid, they may reach an invalid state.

The outbox alone does not solve this. Polling workers can publish in parallel; records ordered by created_at or a generated ID are not necessarily ordered by transaction commit. And Kafka only preserves record order within a partition.

If order matters, define the guarantee precisely: is it per aggregate, per account, per tenant, or globally? Common safeguards include:

Use the same broker partition key for all events concerning an aggregate, such as orderId.
Preserve source ordering before publishing, rather than expecting the broker to repair reordering.
Include an aggregate version or another reliable sequence and reject or buffer stale and out-of-order updates when needed.
Serialize publication or updates per aggregate where the business rules require it.

A partition key alone is not enough if two producers publish updates in the wrong order. Global order is considerably more expensive and usually unnecessary.

Failure scenarios worth testing

Failure	Expected behavior
Business transaction rolls back	No committed outbox event exists.
Application crashes after commit	Event remains in outbox for a relay to publish.
Broker unavailable	Outbox backlog grows; delivery resumes after recovery.
Relay crashes before publishing	Pending event is retried.
Relay crashes after broker acknowledgement	Duplicate publication is possible.
Consumer crashes before its database commit	Delivery can be retried without applying a partial local transaction.
Consumer receives duplicate event	Deduplication prevents a repeated business mutation.
CDC connector stalls	Delivery lag and potentially PostgreSQL WAL retention increase.

Operating an outbox in production

The table is part of your delivery pipeline, not merely an implementation detail. Monitor it like a queue.

Key signals include age of the oldest unpublished event, backlog size, end-to-end publication latency (especially p95/p99), attempted and failed publications, retry counts, and stuck claims.

For CDC, add connector lag, replication-slot health, and retained WAL volume. Alert on growing delivery age rather than only checking that the relay process is alive.

Other decisions are easy to underestimate:

Retention and cleanup: published records must eventually be archived or removed without disrupting relay recovery or CDC replays.
Backpressure: if the broker is down for hours, make sure the database can hold the backlog and the relay can drain it without overwhelming consumers.
Poison events: distinguish transient publication failures from malformed or incompatible events, and provide a replay or remediation procedure.
Schema evolution: publish stable, versioned event contracts. Avoid exposing your internal database structure unintentionally.
Replay: know whether your event ID remains stable across retry and replay, and what consumers will do when they see an old event again.

One subtle point: a business request can be executed more than once because the HTTP client retries after an uncertain response. Transactional Outbox only addresses event publication; it does not automatically make the original business command idempotent. Consider a request-level idempotency key when duplicate commands are dangerous.

When should you use it?

Transactional Outbox is a strong fit when a successful database change must eventually trigger a message and losing that event would violate a business or consistency requirement: order workflows, payment lifecycle events, inventory reservations, account state changes, or important integration events.

It’s less compelling when messages are best-effort telemetry, when there is no database state change to coordinate. It also does not replace a Saga for coordinating multiple services’ business transactions: the outbox ensures reliable publication of each participant’s events; a Saga addresses workflow coordination and compensation.

The trade-off is deliberate. You exchange a fragile synchronous dual-write for an extra table, a delivery process, at-least-once semantics, eventual consistency, monitoring, and consumer idempotency.

The takeaway

The most important lesson is not “use an outbox table.” It is choose the boundary of atomicity carefully.

We usually cannot atomically commit a relational database and a message broker using one ordinary local transaction. But we can atomically commit business state and the intention to publish an event. Once that intention is durable, asynchronous infrastructure can deliver it reliably, recover from crashes, and retry when dependencies are unavailable.

Transactional Outbox does not make distributed systems magically exactly-once or strongly consistent. It makes a specific failure mode manageable and moves the remaining complexity into explicit, testable parts of the architecture.

That’s a trade-off worth understanding before reaching for Kafka, microservices, or another @Transactional annotation.