Marchay Platform

Appearance

Home/Projects/Intelligence/Platform Operations

Intelligence Platform Operations ​

Epic: Crawler Framework Enhancements ​

Plan: Enhance the crawler framework with incremental crawling, change detection, and on-demand re-crawl for stale records.

Architectural Context: Full re-crawls are expensive and slow. Incremental crawling re-crawls only records older than N days, reducing load and improving freshness. Change detection identifies when directory pages have changed, triggering targeted re-crawls. On-demand re-crawl allows manual refresh of stale records.

Tasks ​

  • Implement incremental crawling (re-crawl only records older than N days)
  • Build change detection for directory pages
  • Implement on-demand re-crawl for stale records

Epic: Data Quality Monitoring ​

Plan: Implement per-vertical, per-source coverage metrics and quality regression alerts to maintain data freshness and completeness.

Architectural Context: Data quality is tracked across dimensions: email coverage %, phone coverage %, completeness score, and freshness (average, p50, p95). Alerts fire when quality drops below thresholds (email < 80%, phone < 60%, > 20% stale records), enabling proactive remediation.

Tasks ​

  • Implement per-vertical, per-source coverage metrics (email %, phone %, completeness)
  • Build quality regression alerts (email coverage < 80%, phone < 60%)
  • Implement freshness tracking per-vertical (average, p50, p95)
  • Build stale record alerting (> 20% stale)

Epic: Observability Pipeline ​

Plan: Implement Prometheus metrics, structured logging, per-record tracing, and source health tracking across all crawlers.

Architectural Context: The observability pipeline provides visibility into crawler performance and data quality. Prometheus metrics expose crawl rates and error rates. Structured JSON logging enables correlation by trace_id and source. Per-record tracing tracks deep-crawl extractions. Source health tracking monitors success rate, response time, rate-limit events, and schema changes.

Tasks ​

  • Implement Prometheus metrics endpoint (/metrics)
  • Build structured JSON logging per crawl event
  • Implement per-record tracing for deep-crawl extractions
  • Build source health tracking (success rate, response time, rate-limit events, schema changes)

Epic: Production Deployment ​

Plan: Deploy the Intelligence platform to production with k8s CronJob scheduling, SQLite backups, MCP server transport, and a full CI/CD pipeline.

Architectural Context: Each vertical runs as a k8s CronJob on its own schedule. SQLite databases are backed up to object storage daily with 30-day retention. The MCP server migrates from stdio to HTTP/SSE transport for production use. Health and readiness probes enable k8s orchestration. The CI/CD pipeline enforces quality gates (ruff, mypy, unit tests, integration tests, container build, staging, prod).

Tasks ​

  • Build k8s CronJob scheduling per vertical
  • Implement SQLite backup to object storage (daily, 30-day retention)
  • Build MCP server HTTP/SSE transport (replacing stdio)
  • Implement health and readiness probes
  • Build CI/CD pipeline (ruff, mypy, unit tests, integration tests, container build, staging, prod)

Marchay Platform Documentation

© 2026 Marchay