Deployment

Each service is a container. A minimal production setup:

Container Layout

Service	Replicas	Notes
API Gateway	2+	Stateless, behind load balancer
Orchestrator	2+	Session-sharded, with state in Redis + DB
Redis/Valkey	Managed, or 1 primary + 1 replica	Single point of failure otherwise
Research Agents	N per type	Autoscaling on queue depth

Networking

• API Gateway is the only public-facing service
• All internal traffic (Orchestrator → Redis, workers → Redis, Orchestrator → DB) stays on a private network
• Workers do not need to be reachable from the Orchestrator; they pull, they don't get called

Scaling Policies

• API Gateway: CPU or RPS based
• Research Agents: Queue depth per agent type (e.g., scale Dining workers when LLEN jobs:dining > 100)
• Orchestrator: Manual or session-count based; the bottleneck is usually model latency, not CPU
• Redis: Memory- and connection-count based

Service Definition

Each container exposes:

• A health check (/healthz)
• A readiness check (/readyz) that verifies downstream dependencies
• Prometheus metrics on a standard port