Unified Data Models Build Brief

Build brief for the first epic of Platform Core: Unified Data Models. This is the starting point for all Intelligence Architecture work.

---

Overview

Epic: Unified Data Models Estimated effort: ~1-2 weeks Depends on: Nothing (this is the foundation) Enables: All verticals, Cross-vertical search, Entity resolution

Define shared base types (Location, Contact) used across all verticals, enforce schema consistency, and build a versioning and migration system.

---

Acceptance Criteria

1. Project Setup

• ☐ Initialize marchay-intelligence repo with pyproject.toml
• ☐ Create intelligence/ package structure
• ☐ Set up pytest for testing
• ☐ Set up ruff for linting
• ☐ Set up mypy for type checking
• ☐ Create Docker Compose with SQLite volume

2. Shared Base Types

• ☐ Define Location model:

  class Location(BaseModel):
      id: str  # UUID
      name: str
      address: str | None = None
      city: str
      region: str | None = None
      country: str
      country_code: str  # ISO 3166-1 alpha-2
      postal_code: str | None = None
      latitude: float | None = None
      longitude: float | None = None
      timezone: str | None = None
      created_at: datetime = Field(default_factory=datetime.utcnow)
      updated_at: datetime = Field(default_factory=datetime.utcnow)

• ☐ Define Contact model:

  class Contact(BaseModel):
      id: str  # UUID
      entity_type: str  # 'dmc', 'restaurant', 'hotel', etc.
      entity_id: str
      email: str | None = None
      phone: str | None = None
      phone_e164: str | None = None  # E.164 format
      website: str | None = None
      social: dict[str, str] = {}  # platform -> url
      created_at: datetime = Field(default_factory=datetime.utcnow)
      updated_at: datetime = Field(default_factory=datetime.utcnow)

• ☐ Define VerticalBase model (base for all vertical entities):

  class VerticalBase(BaseModel):
      id: str  # UUID
      vertical: str  # 'dmc', 'dining', 'experiences', etc.
      name: str
      location: Location
      contact: Contact
      source: str  # crawler source
      source_url: str | None = None
      confidence: float = 0.0  # 0-1
      created_at: datetime = Field(default_factory=datetime.utcnow)
      updated_at: datetime = Field(default_factory=datetime.utcnow)

3. Vertical-Specific Models

• ☐ Define DMC model extending VerticalBase:

  class DMC(VerticalBase):
      vertical: Literal['dmc'] = 'dmc'
      services: list[str] = []
      specialties: list[str] = []
      destinations: list[str] = []
      advisor_network: str | None = None
      languages: list[str] = []

• ☐ Define Restaurant model extending VerticalBase
• ☐ Define Experience model extending VerticalBase
• ☐ Define Transportation model extending VerticalBase
• ☐ Define Hotel model extending VerticalBase
• ☐ Define Flight model extending VerticalBase
• ☐ Define SpaWellness model extending VerticalBase

4. Schema Consistency

• ☐ Implement schema validation decorators
• ☐ Build schema registry (vertical -> model mapping)
• ☐ Implement cross-vertical type checking
• ☐ Add schema documentation generation

5. Schema Versioning

• ☐ Implement version tracking per vertical:

  class SchemaVersion(BaseModel):
      vertical: str
      version: str  # semver
      created_at: datetime
      migration_up: str | None = None
      migration_down: str | None = None

• ☐ Build version registry
• ☐ Implement backward compatibility checks
• ☐ Add schema diff utility

6. Migration System

• ☐ Implement Alembic-style migration system for SQLite
• ☐ Build migration generator from model changes
• ☐ Implement up/down migration execution
• ☐ Add migration testing framework

7. Storage Foundation

• ☐ Implement SQLite store with FTS5 support
• ☐ Build base table creation from models
• ☐ Implement CRUD operations for all base types
• ☐ Add JSON export functionality

8. Tests

• ☐ Unit tests for all base models
• ☐ Unit tests for vertical models
• ☐ Unit tests for schema validation
• ☐ Unit tests for versioning
• ☐ Unit tests for migration system
• ☐ Unit tests for SQLite store
• ☐ Integration tests: create, read, update, delete for each vertical

---

File Structure (created by this brief)

marchay-intelligence/
├── intelligence/
│   ├── __init__.py
│   ├── cli.py
│   ├── config/
│   │   ├── __init__.py
│   │   └── settings.py
│   ├── models/
│   │   ├── __init__.py
│   │   ├── base.py
│   │   ├── dmc.py
│   │   ├── dining.py
│   │   ├── experiences.py
│   │   ├── transportation.py
│   │   ├── hotels.py
│   │   ├── flights.py
│   │   └── spa_wellness.py
│   └── storage/
│       ├── __init__.py
│       ├── sqlite_store.py
│       ├── json_store.py
│       └── schema.py
├── tests/
│   ├── __init__.py
│   ├── conftest.py
│   ├── test_models/
│   └── test_storage/
├── migrations/
│   ├── alembic.ini
│   └── versions/
├── pyproject.toml
├── Makefile
└── README.md

---

Implementation Phases

Phase 1: Project Scaffolding (Day 1)

1. Initialize pyproject.toml with dependencies
2. Create package structure
3. Set up pytest, ruff, mypy
4. Create basic CLI entry point
5. Verify python -m intelligence.cli --help works

Phase 2: Base Models (Day 2-3)

1. Implement Location model
2. Implement Contact model
3. Implement VerticalBase model
4. Write tests for all models
5. Verify serialization/deserialization

Phase 3: Vertical Models (Day 3-4)

1. Implement DMC model
2. Implement Restaurant model
3. Implement Experience model
4. Implement Transportation model
5. Implement Hotel model
6. Implement Flight model
7. Implement SpaWellness model
8. Write tests for all vertical models

Phase 4: Schema System (Day 4-5)

1. Implement schema registry
2. Build schema validation
3. Implement version tracking
4. Build schema diff utility
5. Write tests

Phase 5: Storage Foundation (Day 5-6)

1. Implement SQLite store with FTS5
2. Build table creation from models
3. Implement CRUD operations
4. Implement JSON export
5. Write integration tests

Phase 6: Migration System (Day 6-7)

1. Implement Alembic-style migrations
2. Build migration generator
3. Implement up/down execution
4. Write migration tests

Phase 7: Polish (Day 7)

1. Add docstrings and type hints
2. Run all tests
3. Fix lint and type errors
4. Create README.md

---

Dependencies

Dependency	Version	Purpose
pydantic	^2.7.0	Data models, validation
pydantic-settings	^2.3.0	Configuration
typer	^0.12.0	CLI framework
httpx	^0.27.0	Async HTTP client
beautifulsoup4	^4.12.0	HTML parsing
aiosqlite	^0.20.0	Async SQLite
alembic	^1.13.0	Database migrations
pytest	^8.2.0	Testing
pytest-asyncio	^0.23.0	Async test support
ruff	^0.4.0	Linting
mypy	^1.10.0	Type checking