Scaling

Component	Scaling Model	Notes
API Gateway	Stateless, horizontal	Scale on RPS
Orchestrator	Stateful, sharded by session	Sticky routing on session_id; one orchestrator per active conversation
Redis/Valkey	Single primary + replicas (or cluster)	Not a bottleneck at this scale; cluster if queue depth requires
Research Agents	Stateless, horizontal	Each agent type scales independently on queue depth

Orchestrator Scaling

The Orchestrator is the only stateful component that isn't trivially horizontally scalable. Two options:

Sharded by Session

A consistent hash on session_id maps to an orchestrator instance. Simpler, but requires handoff logic if an instance dies.

Stateless Orchestrator + External State Store

All session state lives in Redis. Any orchestrator instance can pick up any session. More moving parts, but easier scaling and recovery.

The second is usually the right call once you have more than a handful of orchestrator instances.

Research Agent Scaling

Each agent type has its own queue. Workers autoscale on queue depth. The Dining agent and the Hotels agent can scale completely independently based on actual demand.

A research agent with a slow downstream API (e.g., a partner reservation system) can be scaled up without affecting faster agents.

Redis Sizing

At expected load (hundreds of concurrent sessions, tens of thousands of jobs/hour), a single primary + replica handles this comfortably. Move to cluster mode if you need horizontal queue partitioning across nodes.