Failure Recovery
Lease expiry, worker loss, fencing, checkpoint validation, idempotency and human review for uncertain side effects.
Detect and reconcile
Heartbeat loss first makes a worker suspect and stops new placement; sustained loss marks it lost according to configurable thresholds. When its lease expires, the controller reconciles task and attempt state before reassignment.
Fencing prevents stale writes
Every new attempt receives a newer generation. Only the currently valid generation may update authoritative task state or complete an attempt. A disconnected worker can keep running locally, but it must not overwrite a newer assignment.
Checkpoint validity
A checkpoint is versioned state with schema, artifact checksum, cursor and compatibility with an agent definition. Resume validates these properties; restarting a process is not automatically a safe continuation. Browser checkpoints may be invalid when the page or session has changed.
External side effects
Use provider idempotency keys and reconcile provider status where supported. An uncertain mutation moves to recovery-pending or human review. Exactly-once effects cannot be guaranteed when external systems provide no idempotency or reconciliation mechanism.
Recovery matrix
- Queued work without a lease: schedule on a suitable worker.
- Pure/idempotent running task: expire lease, then retry or resume a verified checkpoint.
- Read-only external task: retry within rate limits and deduplicate evidence.
- Mutation with unknown result: reconcile before retry; request review when uncertain.
- Non-retryable or high-risk task: hold for an authorized reviewer.