Failure Modes

Failure	Detection	Recovery
Worker dies mid-job	Job unacked, Redis re-delivers after visibility timeout	Orchestrator's job tracker times out; re-enqueues or surfaces partial result
Orchestrator dies mid-session	Next request hits a new instance	New instance reconstructs state from DB/Redis, resumes
Result never arrives	Per-job timeout in Orchestrator	Re-dispatch to another worker, or proceed with partial results
Redis unavailable	Health check	Total system stop. Run as managed service with replicas.
Slow worker	Per-job latency SLO	Timeout, re-dispatch, or proceed with partial results
Bad worker output	Schema validation on result	Orchestrator rejects, re-dispatches with refined brief

Orchestrator Recovery in Detail

When an Orchestrator instance dies:

1. The session's sticky routing key resolves to a different instance
2. The new instance loads session state from the database
3. It checks Redis for any in-flight or completed-but-unread worker results
4. It resumes the conversation with the user's next message

This works because the Orchestrator is designed to reconstruct itself from external state on every turn. In-memory state is a cache, not the source of truth.

Partial Results

The Orchestrator should be willing to return a partial answer with explicit gaps ("I found two great dinner options but couldn't confirm a third within your budget — want me to relax the price or expand the neighborhood?") rather than blocking on a failed worker. This requires a result schema that supports partial success.