Crash After Side Effect

1. Core problem: the process dies while the effect may survive

In this failure family, a worker can commit a durable or external side effect and crash before it records a completion marker or returns an acknowledgement. After restart, the caller only knows that completion was not observed; that does not prove the effect did not happen. The first rule is: do not equate missing acknowledgement with missing effect.

2. Record four boundaries separately: intent, effect, marker, acknowledgement

For each operation, preserve at least four independent boundaries: was intent persisted, is there side-effect evidence, was a completion marker persisted, and did the caller observe acknowledgement? Compressing them into one status hides crashes between boundaries and destroys the evidence needed for safe recovery.

3. Dual writes create a real crash boundary

When business state and an event/effect are written independently, a crash between the writes can leave inconsistent state. Microsoft’s Azure transactional-outbox sample demonstrates this dual-write anomaly and writes the state change plus event in one atomic batch within one logical partition before relaying asynchronously. [AZURE-OUTBOX] That strengthens the stated transaction boundary; it does not make unrelated external effects globally atomic.

4. Crash before effect and crash after effect need different recovery

If reliable evidence shows the effect never occurred, retry with the same operation identity can be defensible. If the effect happened but the marker is missing, blind retry can duplicate the business effect. Recovery must therefore begin from the crash boundary, not from the generic symptom “job failed.”

5. UNKNOWN is a valid state

Sometimes intent is present and the worker crashed, but the effect ledger or remote oracle is unavailable. The outcome must remain UNKNOWN. Filling the gap with “probably not applied” or “must have applied” is not evidence. A good record preserves uncertainty and names the observation that could resolve it.

6. Idempotency identity helps only when semantic intent is stable

AWS describes client request identifiers/idempotency tokens as a way to recognize repeated requests and avoid additional side effects during retries. [AWS-IDEMPOTENT] Reusing the same key for a different intent must not be silently deduplicated; it should conflict. A key without semantic equivalence is not a universal retry license.

7. A successful retry does not prove exactly-once execution

A green retry only proves that the retry path succeeded in this observation. You still need to know whether the earlier effect existed and whether a duplicate was created. A successful response is not causal proof, so this lesson evaluates effect count and reconciliation evidence separately from response state.

8. Transactional outbox has a bounded atomicity claim

An outbox keeps the state change and event record inside one transaction and relays the event later. Azure explicitly bounds the atomic commit to the transactional batch and treats downstream delivery as at-least-once. [AZURE-OUTBOX] Downstream consumers can still need deduplication or idempotency.

9. Reconcile before blind retry

When the effect may already exist, first query the effect ledger or remote service by operation ID or another domain identity. The important outcomes are: no effect exists, exactly one matching effect exists, or evidence is insufficient. Only then choose retry, marker repair, or compensation.

10. Completion-marker evidence matters, but it is not always the effect oracle

A marker can show that a workflow crossed one boundary, but if the marker and external effect are not atomic, marker absence does not prove effect absence. Conversely, a present marker with a lost acknowledgement usually means the caller should reconcile the existing result rather than repeat the business effect.

11. Compensation is not retry

Retry tries to drive the intended operation to completion; compensation applies a separate domain action to offset an already accepted effect. If the domain supports compensation, record its operation identity, evidence, and residual risk separately. Compensation is not a universal rollback guarantee.

12. Negative control: response loss without a process crash

If both the remote effect and completion marker exist and only the network response is lost, the primary family is Network Partition/response loss, not Crash After Side Effect. This negative control prevents labeling every missing response as an internal crash.

13. Transfer the side effect to a separate service

In the transfer fixture, the effect is stored by a separate local service simulation while the worker’s completion marker is delayed. You can no longer assume the local marker and effect share an atomic boundary. Reconcile the remote effect by operation ID, then repair local completion state.

14. Lab: eight fixtures and 64 assertions

The lab is fully local, deterministic, and synthetic. Eight cases cover a healthy baseline, crash-before-effect, crash-after-effect-before-marker, crash-after-marker-before-ack, idempotency conflict, network-response-loss negative control, incomplete evidence, and cross-service transfer. The hidden evaluator checks eight outputs per case for 64 assertions total.

15. Exit checklist and claim boundary

To exit, you must separate intent/effect/marker/acknowledgement; classify the crash boundary from evidence; preserve UNKNOWN; choose retry only after reconciliation/idempotency reasoning; reject same-key/different-intent reuse; bound the outbox claim to its actual atomic boundary; rebuild the transfer case from new evidence; and state that the synthetic fixture does not prove global exactly-once execution.