Intelligence Project Structure

Separate repository (marchay-intelligence) for all Intelligence Architecture services. This document defines the repo layout, tech stack, and conventions.

---

Repository Layout

marchay-intelligence/
├── intelligence/                   # Main package
│   ├── __init__.py
│   ├── cli.py                      # CLI orchestrator (typer)
│   ├── config/
│   │   ├── __init__.py
│   │   ├── settings.py             # Settings via pydantic-settings
│   │   └── sources.yaml            # Per-vertical source config
│   │
│   ├── models/                     # Pydantic data models
│   │   ├── __init__.py
│   │   ├── base.py                 # Shared base types (Location, Contact)
│   │   ├── dmc.py                  # DMC vertical models
│   │   ├── dining.py               # Dining vertical models
│   │   ├── experiences.py          # Experiences vertical models
│   │   ├── transportation.py       # Transportation vertical models
│   │   ├── hotels.py               # Hotels vertical models
│   │   ├── flights.py              # Flights vertical models
│   │   └── spa_wellness.py         # Spa & Wellness vertical models
│   │
│   ├── crawlers/                   # Per-vertical crawlers
│   │   ├── __init__.py
│   │   ├── base.py                 # Abstract BaseCrawler (shared)
│   │   ├── dmc/
│   │   │   ├── __init__.py
│   │   │   ├── inside_travel.py
│   │   │   ├── traveller_made.py
│   │   │   └── dmc_travel.py
│   │   ├── dining/
│   │   │   ├── __init__.py
│   │   │   ├── resy.py
│   │   │   ├── opentable.py
│   │   │   ├── yelp.py
│   │   │   └── tock.py
│   │   ├── experiences/
│   │   │   ├── __init__.py
│   │   │   ├── viator.py
│   │   │   ├── getyourguide.py
│   │   │   └── klook.py
│   │   ├── transportation/
│   │   │   ├── __init__.py
│   │   │   ├── uber.py
│   │   │   ├── lyft.py
│   │   │   └── blacklane.py
│   │   ├── hotels/
│   │   │   ├── __init__.py
│   │   │   ├── booking.py
│   │   │   ├── expedia.py
│   │   │   └── virtuoso.py
│   │   ├── flights/
│   │   │   ├── __init__.py
│   │   │   ├── amadeus.py
│   │   │   ├── skyscanner.py
│   │   │   └── google_flights.py
│   │   └── spa_wellness/
│   │       ├── __init__.py
│   │       ├── mindbody.py
│   │       └── spafinder.py
│   │
│   ├── extractors/                 # Shared extraction logic
│   │   ├── __init__.py
│   │   ├── contact.py              # Email/phone/social extraction
│   │   ├── website.py              # Deep-crawl extraction
│   │   └── structured.py           # Structured data extraction
│   │
│   ├── storage/                    # Per-vertical storage
│   │   ├── __init__.py
│   │   ├── sqlite_store.py         # SQLite + FTS5 (shared)
│   │   ├── json_store.py           # JSON export (shared)
│   │   ├── migrations/             # Schema migrations
│   │   │   ├── __init__.py
│   │   │   └── versions/
│   │   └── schema.py               # Schema definitions
│   │
│   ├── mcp_server/                 # MCP server
│   │   ├── __init__.py
│   │   ├── server.py               # MCP server entry point
│   │   ├── tools/                  # Per-vertical MCP tools
│   │   │   ├── __init__.py
│   │   │   ├── dmc.py
│   │   │   ├── dining.py
│   │   │   ├── experiences.py
│   │   │   ├── transportation.py
│   │   │   ├── hotels.py
│   │   │   ├── flights.py
│   │   │   └── spa_wellness.py
│   │   └── resources/              # MCP resources
│   │       ├── __init__.py
│   │       └── verticals.py
│   │
│   └── cross_vertical/             # Cross-vertical features
│       ├── __init__.py
│       ├── search.py               # search_destination tool
│       ├── entity_resolver.py      # Entity resolution
│       └── itinerary.py            # suggest_itinerary tool
│
├── tests/
│   ├── __init__.py
│   ├── conftest.py
│   ├── test_models/
│   ├── test_crawlers/
│   ├── test_extractors/
│   ├── test_storage/
│   ├── test_mcp/
│   └── test_cross_vertical/
│
├── migrations/                     # Alembic migrations
│   ├── alembic.ini
│   └── versions/
│
├── docker/
│   ├── Dockerfile
│   └── docker-compose.yml
│
├── pyproject.toml
├── Makefile
├── .gitlab-ci.yml
└── README.md

---

Tech Stack

Component	Choice	Rationale
Language	Python 3.12+	Proven in spike, LangChain ecosystem
CLI	typer	Fast, type-safe CLI
Models	Pydantic v2	Validation, serialization, JSON Schema
Storage	SQLite + FTS5	Lightweight, full-text search, no infra
Crawlers	httpx + BeautifulSoup	Async HTTP, HTML parsing
JS Rendering	Playwright	For JS-rendered pages
MCP Server	mcp (Python SDK)	Official MCP implementation
Testing	pytest + pytest-asyncio	Async test support
Linting	ruff	Fast, comprehensive
Type Checking	mypy (strict)	Catch errors early
Migrations	Alembic	Standard for SQLite/Postgres + Python

---

Naming Conventions

• Python: snake_case for files, functions, variables. PascalCase for classes.
• Files: base_crawler.py, contact_extractor.py, test_store.py
• Tables: snake_case plural (dmc, restaurant, experience)
• MCP Tools: search_{vertical}, get_{vertical}_details, list_{vertical}_types
• Metrics: intelligence_crawler_requests_total, intelligence_records_created_total
• Git branches: feat/dmc-crawler, fix/resy-rate-limit, chore/update-deps

---

Repo Relationship

This repo (marchay-intelligence) is separate from the architecture docs repo (marchay-docs) and other service repos.

Repo	Purpose	Contents
marchay-docs	Architecture planning	Design docs, build briefs, roadmaps
marchay-agent	Agent services	Orchestrator, research agents
marchay-integration	Integration services	API Gateway, connectors
marchay-intelligence	Intelligence services	Crawlers, storage, MCP server

The build brief in marchay-docs serves as the spec. Implementation lives in marchay-intelligence.

---

Makefile

.PHONY: dev test lint typecheck crawl migrate

dev:
	python -m intelligence.cli --help

test:
	pytest tests/ -v --cov=intelligence --cov-report=term-missing

lint:
	ruff check intelligence/ tests/

typecheck:
	mypy intelligence/

crawl:
	python -m intelligence.cli crawl --vertical $(VERTICAL) --source $(SOURCE)

migrate:
	alembic upgrade head

build:
	docker compose build

prod:
	docker compose -f docker-compose.prod.yml up -d