The request is not the whole world
Some work completes immediately. Some work has to wait for a human, a database, a remote API, or a scheduled retry. Treating all of that as one long request is how systems become fragile.
Three simple ideas help:
- An event says something happened:
checkout.error-rate-crossed. - A queue holds work until a worker can take it.
- A workflow coordinates steps, state and outcomes over time.
Our incident flow
alert received
→ create investigation
→ run deterministic checks
→ record evidence
→ request human review if the rules cannot decide
→ close or continue
The Workbench is deliberately small, so it uses local scenario data rather than a real message broker. The mental model is the same. Later, when you meet queues and workflow engines in production, you will recognise the responsibilities.
Retries need a reason
Retries are not a magic word for reliability. A retry is only sensible when the operation is safe to repeat and the failure might be temporary.
For example, reading a metric snapshot can be retried. Charging a customer cannot be casually repeated. This is where idempotency matters: an operation is idempotent when you can run it more than once and leave the system in the same intended final state as one run. A duplicate “create ticket” request should still leave us with one ticket; a duplicate charge must not create a second payment.
The response may say “already exists” on the second attempt. That is fine. The important part is that the operation does not create a second effect. This is about safe retries, not regression testing.
Try it
Choose Delayed metric in the Workbench. Its first metric-snapshot read times out; its retry succeeds. Inspect the event timeline: it records the timeout, the one permitted retry, and the successful result that became evidence.
The scenario does not hide the first failure because a later step worked. Silence is not evidence of success.
Keep this with you
A workflow is not fancy orchestration. It is a promise that work can survive time, waiting and failure without forgetting what it was doing.