Appearance
Intelligence Architecture — Overview
The intelligence platform is our internal data layer for luxury travel knowledge. It crawls public and partner sources across multiple verticals (DMCs, dining, hotels, experiences, transportation, etc.), structures the data, and exposes it to AI agents through a standardized interface (MCP).
The platform is vertical-agnostic by design — the spike validated the pattern with DMCs, and every subsequent vertical reuses the same framework.
What This Is
- A reusable crawler framework with rate limiting, retry, concurrency
- A shared extraction layer (contact info, deep-crawl, structured data)
- A per-vertical storage layer (SQLite + FTS5, JSON export)
- A unified MCP server that exposes data to AI agents
- A CLI for orchestration and ad-hoc operation
What This Is Not
- Not a public-facing travel product
- Not a booking engine (that's downstream)
- Not a partner integration platform (that's the Integrations Architecture)
- Not a single-vertical scraper — it's a framework that ships with verticals
Architecture
intelligence/
├── pyproject.toml
├── crawler.py # CLI orchestrator
├── models.py # Pydantic data models
├── config/
│ └── sources.yaml # Per-vertical source config
├── crawlers/
│ ├── base.py # Abstract BaseCrawler (shared)
│ ├── dmc/ # DMC vertical crawlers
│ ├── dining/ # (planned) Dining vertical
│ ├── hotels/ # (planned) Hotels vertical
│ └── ... # (planned) other verticals
├── extractors/
│ ├── contact.py # Email/phone/social (shared)
│ └── website.py # Deep-crawl (shared)
├── storage/
│ ├── sqlite_store.py # SQLite + FTS5 (shared, per-vertical schema)
│ └── json_store.py # JSON export (shared)
└── mcp_server/
└── server.py # MCP server (shared pattern, per-vertical tools)
Reference Implementation: DMC Spike
The DMC vertical was built in ~3 hours and validated end-to-end:
- 3 source crawlers (Inside.travel, Traveller Made, DMC.travel)
- 231 DMCs indexed, 95% email coverage, 80% phone coverage
- 150+ destinations, 13 service categories, 52 specialties
- 5 MCP tools, 3 resources, 2 prompt templates
The spike proved:
- The BaseCrawler pattern is sufficient for directory-style sources
- Contact extraction works across many heterogeneous business sites
- SQLite + FTS5 is the right storage choice at this scale
- MCP is a clean interface for AI agent consumption
How Agents Consume This
Our agents (Orchestrator, Workers, etc.) call the MCP server to look up travel intelligence:
Agent (via Atlantes)
→ MCP client
→ MCP server (intelligence platform)
→ search_dmc(destination="Greece", service="Luxury Travel")
→ structured DMC record
The MCP server is the contract. The crawlers, extractors, and storage are implementation details that can evolve.
Vertical Roadmap
| Vertical | Status | Data Sources | LOE |
|---|---|---|---|
| DMCs | Spike Complete | Inside.travel, Traveller Made, DMC.travel | 50–70 hrs |
| Dining | Planned | Resy, OpenTable, Yelp, Tock | 40–56 hrs |
| Experiences | Planned | Viator, GetYourGuide, Klook | 68–98 hrs |
| Transportation | Planned | Uber, Lyft, Blacklane, rental APIs | 76–100 hrs |
| Hotels | Planned | Booking.com, Expedia, Virtuoso | 40–60 hrs |
| Flights | Planned | Amadeus, Skyscanner, Google Flights | 30–50 hrs |
| Spa & Wellness | Planned | Mindbody, SpaFinder | 16–24 hrs |
| Event Tickets | Planned | Ticketmaster, Eventbrite | 16–24 hrs |
| Platform Core | Planned | Shared infrastructure | 40–60 hrs |
| Total | ~460 hrs (~13 weeks) |
See Verticals for the full breakdown.