Observability

The platform is data infrastructure, and data infrastructure is only useful if you trust it. Observability is how we maintain that trust.

Data Quality Metrics

Per-vertical, per-source:

Metric	Definition	Alert Threshold
Total records	Count of indexed entities	n/a
Email coverage %	Records with at least one email	< 80%
Phone coverage %	Records with at least one phone	< 60%
Description coverage %	Records with non-empty description	< 90%
Freshness (avg days since crawl)	Mean of crawled_at age	> 30 days
Stale record %	Records older than threshold	> 20%

DMC Spike Numbers (Baseline)

• Total: 231
• Email: 95%
• Phone: 80%

These are our targets. If a re-crawl drops email coverage below 80%, something is wrong.

Crawl Health

Per run:

Metric	Definition	Alert Threshold
Run duration	Time to complete	> 2× median
Records processed	Total touched	n/a
Records added	New since last run	n/a
Records updated	Changed since last run	n/a
Error count	Failures during the run	> 5% of records
Source availability	Was the source reachable?	< 99% over 24h

Source Health

Per source, rolling 24h / 7d / 30d:

• Request success rate
• Average response time
• Rate-limit events
• Schema change events (e.g., new fields appearing in HTML, fields disappearing)

A source that has been quietly changing its HTML structure is a source that's about to break our crawler. The platform detects this and alerts.

Freshness

Every record has crawled_at. The platform computes:

• Per-vertical freshness — average, p50, p95
• Stale record count — records older than the per-vertical threshold
• Per-source freshness — which sources are stale

Freshness is exposed as an MCP tool (catalog_stats) and a metric.

Logging

Structured JSON, one log line per significant event:

{
  "timestamp": "...",
  "level": "info",
  "event": "crawl.completed",
  "vertical": "dmc",
  "source": "inside_travel",
  "run_id": "uuid",
  "records_processed": 192,
  "duration_seconds": 342,
  "errors": 0
}

Indexed by vertical, source, run_id.

Tracing

Per-record traces for deep-crawl extractions:

• crawl.list.fetch — fetching a directory page
• crawl.detail.fetch — fetching a single entity's page
• extract.contact — extracting contact info from a page
• storage.upsert — writing to SQLite

Useful for diagnosing "why is record X missing email Y?"

Metrics Endpoint

The platform exposes Prometheus metrics on a standard port:

GET /metrics

Including:

• crawler.runs.total{vertical, source, status}
• crawler.records.processed{vertical, source}
• crawler.errors.total{vertical, source, error_type}
• storage.records.total{vertical}
• storage.coverage{vertical, field} (gauge, 0.0–1.0)
• storage.freshness.days{vertical, percentile}

Alerts

• Per-source error rate > 5% over 1h
• Per-vertical data coverage regression (email %, phone %, etc.)
• Crawl run duration > 2× median
• Source unreachable for > 1h
• Storage disk usage > 80%
• MCP server unreachable