Agent System Scaling and Reliability

---

Epic: Orchestrator Scaling

Plan: Scale the Orchestrator horizontally using session sharding with consistent hashing. Support both stateful (sticky sessions) and stateless (external state in Redis) deployment modes. The bottleneck is usually model latency, not CPU, so scaling is manual or session-count based.

Architectural Context: The Orchestrator is the only stateful component that is not trivially horizontally scalable. Two options: (1) Sharded by Session with consistent hash on session_id, or (2) Stateless Orchestrator + External State Store in Redis where any instance can pick up any session. Option 2 is the right call once you have more than a handful of instances.

Tasks

• Implement consistent hash ring on session_id for Orchestrator sharding
• Build session affinity routing: same session_id → same Orchestrator instance
• Implement stateless Orchestrator option: all session state in Redis, any instance can handle any request
• Build Orchestrator instance health checks (HTTP /healthz)
• Implement instance removal from hash ring on failure
• Build session migration: reassign session to healthy instance on failure
• Add load testing: validate shard distribution under concurrent sessions
• Implement scaling metrics: sessions per instance, request latency per instance
• Build scaling playbook: when and how to add/remove Orchestrator instances
• Add circuit breaker: if Redis unavailable, fail fast with clear error message

---

Epic: Research Agent Autoscaling

Plan: Scale Research Agent workers independently based on queue depth per agent type. Each vertical scales based on its own demand signal. A research agent with a slow downstream API can be scaled up without affecting faster agents.

Architectural Context: Each agent type has its own queue (jobs:dining, jobs:experiences, etc.). Workers autoscale on queue depth: scale up when LLEN jobs:{type} > threshold, scale down when queue depth < threshold for N minutes. Min/max limits prevent over/under-scaling.

Tasks

• Implement queue depth monitoring per agent type (via Redis LLEN)
• Build autoscaling policy: scale up when LLEN jobs:{type} > 100 for > 2 minutes
• Build scale-down policy: scale down when LLEN jobs:{type} < 10 for > 10 minutes
• Implement min/max worker limits per agent type (min: 2, max: 20)
• Build worker registration: workers register with Redis on startup, deregister on shutdown
• Implement cooldown period: 5 minutes between scaling events
• Build scaling metrics: workers per agent type, scaling events, queue depth over time
• Add scaling alerting: scale-up events, max workers reached, min workers reached
• Implement cost tracking: correlate scaling events with cloud spend
• Build scaling dashboard: queue depth, worker count, latency over time

---

Epic: Failure Mode Handling

Plan: Handle all failure scenarios gracefully: worker crashes, Orchestrator failures, bad outputs, timeouts, and Redis unavailability. Ensure no jobs are lost and users get responses even with partial data.

Architectural Context: The system is designed for graceful degradation. Workers that die mid-job are detected via visibility timeout and jobs are re-delivered. Orchestrator failures are recovered by reconstructing state from DB/Redis. Bad worker outputs are caught by schema validation and re-dispatched with refined briefs. Partial results are surfaced to users with explicit gaps.

Tasks

• Implement worker death recovery:
• Set visibility timeout on job dequeue (e.g., 60 seconds)
• If worker doesn't ack within timeout, job returns to queue
• Track worker death rate for alerting
• Build Orchestrator death recovery:
• Next request hits new instance via consistent hash
• New instance reconstructs state from Redis/Postgres
• Check for in-flight or completed-but-unread worker results
• Resume conversation with user's next message
• Implement per-job timeout:
• Configurable per agent type (dining: 30s, experiences: 60s, flights: 45s)
• On timeout: re-dispatch to another worker, or proceed with partial results
• Build bad worker output handling:
• Schema validation on result (must match expected output shape)
• Reject invalid results, log warning
• Re-dispatch with refined brief (add constraints, specify expected format)
• Design partial result schema:

{
  "status": "partial",
  "data": [...],
  "confidence": 0.7,
  "gaps": ["Could not confirm availability for 3rd option"],
  "sources": ["opentable", "resy"]
}

• Build circuit breaker for repeated worker failures:
• Open circuit if > 50% jobs fail for an agent type in 1 minute
• Half-open: try 1 job every 30 seconds
• Close circuit on success
• Implement dead-letter queue analysis:
• Periodic job to analyze dead-letter patterns
• Alert on recurring failure modes
• Build Redis failover:
• If Redis unavailable, system stops (documented limitation)
• Implement Redis Sentinel or managed service for automatic failover
• Add failure metrics: worker death rate, Orchestrator reconstruction rate, timeout rate, bad output rate, circuit breaker state