Agent Observability and Operations

---

Epic: Distributed Tracing

Plan: Implement OpenTelemetry tracing across the entire agent system. Each user turn gets a trace ID that follows the request from API Gateway → Orchestrator → tool cascade → job dispatch → worker execution → synthesis. A single user turn should be reconstructable end-to-end from one trace ID.

Architectural Context: The Orchestrator emits a parent span for the whole interaction. Worker jobs are child spans under the parent. Tool cascade calls are sub-spans within the Orchestrator's span. This enables end-to-end debugging of any request, from user message to final response.

Tasks

• Set up OpenTelemetry collector (Grafana Tempo)
• Implement trace context propagation (trace_id, span_id) across services
• Build Orchestrator parent span: one span per user turn, attributes include session_id, member_id, turn_number
• Build tool cascade child spans: one span per tier (Tier 1, Tier 2, Tier 3), attributes include tool_name, latency, success
• Build job dispatch child span: one span per subtask, attributes include job_id, agent_type, task
• Build worker execution child span: one span per job execution, attributes include agent_type, job_id, llm_calls, tool_calls
• Build synthesis child span: one span for result synthesis, attributes include input_count, output_tokens
• Implement span attributes: job_id, agent_type, session_id, member_id, model_used, token_count
• Build trace visualization: Grafana Tempo UI for trace exploration
• Implement trace sampling: 100% for errors, 10% for success (configurable)
• Add trace-based alerting: alert on trace duration > threshold
• Build trace export: export traces to long-term storage (S3/GCS) for analysis

---

Epic: Metrics and Alerting

Plan: Implement comprehensive metrics for queue health, job latency, worker performance, and system reliability. Build alerting for SLO violations and anomalies. Use Prometheus for metrics collection and Grafana for visualization.

Architectural Context: Key metrics from the architecture: queue depth per agent type (autoscaling signal), job latency (primary SLO), worker p50/p95/p99 latency (per-agent SLO), tool cascade call count (cost tracking), synthesis token count (cost tracking), schema validation failures (quality signal).

Tasks

• Implement Prometheus metrics endpoint on all services (/metrics)
• Build queue metrics:
• agent_queue_depth{agent_type} (gauge) — current queue depth per agent type
• agent_jobs_enqueued_total{agent_type} (counter) — total jobs enqueued
• agent_jobs_completed_total{agent_type} (counter) — total jobs completed
• agent_jobs_failed_total{agent_type} (counter) — total jobs failed
• Build latency metrics:
• agent_job_latency_seconds{agent_type} (histogram) — enqueue to completion
• agent_worker_latency_seconds{agent_type, percentile} (summary) — worker execution time
• Build LLM metrics:
• agent_llm_calls_total{phase} (counter) — decompose, synthesize, worker calls
• agent_llm_tokens_total{phase, direction} (counter) — input/output tokens per phase
• agent_llm_cost_dollars{phase} (counter) — cost per phase
• Build quality metrics:
• agent_schema_validation_failures_total{agent_type} (counter)
• agent_replan_rate (gauge) — re-planning events per minute
• Implement alerting rules:
• Queue depth > 100 for > 5 minutes (critical)
• Job p95 latency > SLO (warning)
• Re-plan rate > 10/minute (warning — brief quality issue)
• Redis connection errors > 0 (critical)
• Schema validation failure rate > 5% (warning)
• Build Grafana dashboards:
• System overview: queue depth, job throughput, latency distribution
• Per-agent view: queue depth, latency, success rate per agent type
• Cost view: LLM spend by phase, token usage trends
• Implement alerting integration via Grafana (unified alerting for all thresholds)
• Add runbook links in all alert descriptions

---

Epic: Cost Telemetry

Plan: Track LLM and API costs per turn, broken down by phase (context loading, dispatch, synthesis). Build cost attribution dashboards to identify optimization opportunities. The principle: "You can't optimize what you can't attribute."

Architectural Context: The Orchestrator should log the per-turn token cost and tool call cost, broken down by phase (context load, dispatch, synthesis). This enables cost attribution by member, trip, agent type, and phase. Cost optimization opportunities include: caching frequent tool calls, using cheaper models for low-complexity tasks, reducing re-plan rate.

Tasks

• Implement token counting per LLM call: count input + output tokens
• Build cost calculation: multiply tokens by model pricing (GPT-4: $30/1M input, $60/1M output; Claude: varies)
• Implement per-turn cost aggregation: sum costs across decompose, dispatch, synthesis phases
• Build cost breakdown by phase:
• Context load: Tier 1 + Tier 2 + Tier 3 tool calls
• Dispatch: LLM decompose call
• Synthesis: LLM synthesize call
• Implement cost attribution by dimension:
• Per member (highest spenders)
• Per trip (complex vs. simple trips)
• Per agent type (which verticals are most expensive)
• Per phase (where is the money going)
• Build cost dashboard:
• Total cost over time
• Cost per turn distribution
• Cost by phase breakdown
• Cost by agent type breakdown
• Cost per member (top 20)
• Add cost alerting:
• Cost per turn > $0.50 (warning)
• Total daily cost > $100 (critical)
• Cost spike > 2x baseline (warning)
• Implement cost optimization recommendations:
• Cache frequent tool calls (reduce Tier 1/2/3 calls)
• Use cheaper models for low-complexity tasks
• Reduce re-plan rate (improve brief quality)
• Batch similar requests