Make partial progress visible
Give each meaningful step a durable status such as pending, running, succeeded, or failed, along with a useful error and correlation ID. This helps operators distinguish a transient interruption from a completed action whose response was lost.
Before retrying, determine whether the step is idempotent. If it sends a message, charges an account, or creates a resource, use a stable idempotency key or check for the existing result so a retry does not duplicate the effect.
Resume from a known checkpoint
Persist the data required to restart from the failed step and verify that earlier outputs are still valid. If upstream data has changed, rerun dependent steps deliberately instead of combining results from different versions.
Set bounded retry rules with delays for temporary failures, then route exhausted or permanent failures to an actionable review queue. Alert with the affected run and next recovery action; do not hide a failed run behind an overall success status.