Network Partition

1. A timeout is an observation, not a causal verdict

In a distributed system, a caller can time out because the request never arrived, because it arrived but the response did not return, or because the remote operation completed while its result became unknowable to the caller. AWS explicitly discusses this UNKNOWN outcome boundary for network operations. [AWS-DISTSYS] The first rule of this lesson is therefore: do not equate timeout with remote failure or remote non-execution.

2. Model a partition as a reachability matrix

Instead of one vague label such as “the network is down,” record directions separately: client→service, service→client, service A→service B, and probe paths. A partition may be symmetric or asymmetric. A can observe B while B cannot return a response to A. The record must preserve direction for every observation.

3. Remote outcome may legitimately remain UNKNOWN

If the caller sent a write and timed out, you cannot conclude that the write did not happen without a ledger, idempotency key, remote trace, or equivalent evidence. The AWS Builders’ Library describes this exact uncertainty boundary. [AWS-DISTSYS] In this lesson, UNKNOWN is a valid evidence state, not analytical failure.

4. Separate timeout, deadline, and retry budget

A timeout bounds one attempt; a deadline bounds the total operation; a retry budget bounds the number or cost of further attempts. If these three are designed independently, a small outage can grow into queueing, thread exhaustion, or cascading load.

5. Retry is useful, but it can amplify an outage

AWS recommends bounded retries with backoff and jitter because excessive retries can increase load on an already unhealthy dependency. [AWS-RETRY] When an operation has side effects, retry without idempotency can duplicate those effects. “Timed out, so retry” is not a complete policy.

6. Finite retry and circuit breaking are containment boundaries

Azure guidance likewise recommends timeouts, finite retry strategies, and circuit breaking for persistent failure. [AZURE-TRANSIENT] A circuit breaker does not guarantee availability or correctness; it is a containment mechanism that prevents unbounded attempts and gives a dependency room to recover.

7. Do not confuse service failure with network partition

If reachability is healthy and the service returns a definite 5xx or application error, evidence points more directly to service failure than partition. This lesson includes a negative control so the learner must distinguish “the request failed” from the actual failure family.

8. Asymmetric partition is easy to hide behind green dashboards

A health check from A can be green while the return path or B→A path is broken. An aggregate metric cannot prove direction. Partition diagnosis therefore needs at least two independent directional observations.

9. Degraded mode needs an explicit write boundary

During a partition, read-only degraded mode may be acceptable, while stale reads or writes against an old replica can create inconsistency. If the consistency requirement is unknown, do not silently allow writes. Containment must state which operations fail fast, queue, degrade, or become write-fenced.

10. Recovery is not complete when ping starts working

Connectivity restoration is only the beginning of recovery. You still need recovery probes, state reconciliation, and convergence evidence. A stale replica can be reachable but unsafe. Closing the incident before convergence confuses transport recovery with application recovery.

11. Measure retry amplification

The fixture makes retry budget explicit. If many callers time out together and retry in sync, load can multiply. Backoff and jitter are intended to reduce that correlation. [AWS-RETRY] The record must capture retry count, budget exhaustion, and breaker/degraded state.

12. Actually reject an alternative hypothesis

At least one plausible alternative must be rejected with negative evidence: service crash, application rejection, local client failure, or stale replica without a partition. Writing “probably the network” is insufficient. Classification must connect to directional reachability and remote-effect evidence.

13. Closure requires independent convergence proof

After connectivity returns, do not merely rerun the same timed-out request. Recheck reachability, verify remote effect or ledger state, reconcile stale state, and then execute an independent request. If convergence is not proven, retain RECOVERY_NOT_CONVERGED or OPEN.

14. Lab: eight fixtures and 64 assertions

The lab is fully local and deterministic. Eight cases cover healthy baseline, symmetric request drop, response loss with remote effect, one-way partition, deadline delay, retry-budget exhaustion, adjacent service failure, incomplete evidence, and transfer with response loss plus stale replica state. The hidden evaluator checks eight outputs per case for 64 assertions total.

15. Exit checklist and claim boundary

To pass, you must build a reachability matrix; separate timeout from remote outcome; preserve UNKNOWN; bound retry/deadline/breaker behavior; reject one alternative; distinguish restoration from convergence; rebuild the transfer case from new evidence; and state explicitly that success in this fixture proves only the bounded reasoning exercised here, not the behavior of every network or distributed system.