Durable workflow engines are not queues

A queue delivers a message. A workflow engine has to remember what happened after the message was delivered, what the step returned, and whether the side effect it caused is allowed to happen again. Those are different problems, and treating the second as the first is how systems end up with duplicate payments and half-finished migrations.

A queue gives you at-least-once delivery, not at-most-once effects

Most teams start with a queue because it is the tool in front of them. A message goes in, a worker takes it, and if the worker dies the message returns. That is a useful guarantee. It is also the only guarantee the queue makes.

The moment a step calls an external system, the guarantee stops at the boundary. The worker may have charged the card and died before acknowledging the message. The queue is now obliged to deliver the message again. From the queue’s point of view it is behaving correctly. From the ledger’s point of view the customer has been charged twice.

The fix is not a better queue. It is a record of the effect, written in the same transaction as the state change, with a key the retry can recognise.

Three records, not one

A durable workflow needs three things that a queue does not keep:

• Workflow state — which step is current, what input it received, and what the previous step returned.
• Attempt history — how many times a step has run, why the last attempt failed, and when it is next eligible.
• Effect ledger — the side effects that have already happened, keyed by an idempotency token that survives a retry.

The first two are bookkeeping. The third is the one that protects the business.

CREATE TABLE step_effect (
    workflow_id     uuid        NOT NULL,
    step_name       text        NOT NULL,
    idempotency_key text        NOT NULL,
    effect          jsonb       NOT NULL,
    created_at      timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (workflow_id, step_name, idempotency_key)
);

The primary key is the contract. A retry asks the ledger whether the effect exists before it calls the outside world. If it does, the step returns the recorded result and the workflow advances. The external system is called once, no matter how many times the message is delivered.

Idempotency is a schema decision

It is tempting to put idempotency in application code: a set of recently seen keys, a Redis key with a TTL, a check-then-act sequence. All three fail in the same way. They are not in the same transaction as the effect, so a crash between the check and the write reopens the window the check was supposed to close.

The ledger has to be durable, transactional and unique. If the database is PostgreSQL, that means a real table with a real primary key, written in the same transaction as the workflow state transition. If the effect is in another system, the ledger records the intent before the call and the result after it, and a reconciler closes the gap.

Failure modes worth designing for

The table is the review checklist. If a workflow cannot answer the right-hand column for each row, it is a queue with extra steps.

What we check in review

We ask four questions before a workflow is allowed near a financial path:

1. What is the idempotency key, and what makes it stable across retries?
2. Which transaction writes the effect record, and does it also write the state transition?
3. What happens if the process dies between the external call and the ledger write?
4. Who reconciles the gap, and how do they find it?

If the answer to any of them is “the queue handles it”, the design is not finished. The queue handles delivery. Durability is the part we have to build.