Design Principles

The intelligence platform is built to be vertical-agnostic, AI-first, and operationally boring. These principles are non-negotiable.

1. Vertical-Agnostic Framework, Per-Vertical Data

The crawler, extractor, storage, and MCP patterns are shared. The schemas, sources, and tools are per-vertical. Adding a new vertical means adding a new directory under crawlers/, new tables in SQLite, and new MCP tools — not rewriting infrastructure.

2. AI-First Interface

The MCP server is the primary consumer interface. Every piece of data the platform produces is intended to be queried by an AI agent. Schemas, tool names, and resource formats are designed for LLM consumption first, humans second.

3. Idempotent and Resumable Crawls

Crawls must be safely re-runnable. A re-crawl of the same source with the same config produces the same end state. Partial failures can be resumed from the last successful record. No "start over" failures.

4. Source Failures Don't Break the System

If one source fails (rate limit, anti-bot, schema change), other sources still crawl. The platform tracks per-source health and degrades gracefully.

5. Bounded Aggression

We crawl politely. Rate limits per source are explicit, configurable, and enforced. The platform never overwhelms a source — our reputation as a partner and our ability to keep crawling depend on this.

6. Data Quality Is Measurable

Every vertical tracks coverage metrics (e.g., email %, phone %, completeness score). Quality regressions trigger alerts. We don't ship crawls that produce empty records.

7. Data Freshness Is Tracked

Every record has a crawled_at timestamp. Stale records are flagged. The platform supports incremental re-crawls (only records older than N days).

8. Schema Stability With Versioned Migrations

Schema changes are migrations, not breaking changes. Old records are migrated forward; new fields are nullable with sensible defaults.

9. Pydantic Everywhere

All data flows through Pydantic models. This gives us type safety, auto-serialization, and free schema generation for MCP tools.

10. The Spike's Decisions Are Load-Bearing

Decisions made in the DMC spike — uv, BeautifulSoup, SQLite, MCP, Pydantic, BaseCrawler — are the platform's decisions. New verticals follow them. We don't re-litigate architecture per vertical.