DMC Data Enrichment Build Brief

Build brief for the first epic of the DMC Vertical: DMC Data Enrichment. This scales the DMC dataset from the initial spike (~231 DMCs) to production volume (10,000+ DMCs).

Epic: DMC Data Enrichment

---

Overview

Epic: DMC Data Enrichment Estimated effort: ~2-3 weeks Depends on: Unified Data Models (complete), DMC spike crawlers (complete) Enables: DMC Data Quality, DMC Advanced Extraction, DMC Production Operations

Expand the DMC dataset from the initial spike by integrating DMC Finder (10,000+ DMCs, 120+ countries), and members of ADMEI. Cross-source deduplication ensures a single canonical DMC record per company.

---

Context

What Exists (Spike Complete)

The initial spike validated DMC data extraction with three sources:

Source	DMCs	Status
Inside.travel	~192	Working
Traveller Made	~39	Working
DMC.travel	~0	Parsing issues

Spike results: 231 DMCs indexed, 95% email coverage, 80% phone coverage, 150+ destinations, 13 service categories, 52 specialties.

Existing infrastructure:

• crawlers/base.py — Abstract BaseCrawler with rate limiting, retry, concurrency
• crawlers/dmc/inside_travel.py — Inside.travel crawler
• crawlers/dmc/traveller_made.py — Traveller Made crawler
• crawlers/dmc/dmc_travel.py — DMC.travel crawler (needs work)
• models/dmc.py — DMC model extending VerticalBase
• storage/sqlite_store.py — SQLite + FTS5 storage
• extractors/contact.py — Email/phone/social extraction
• extractors/website.py — Deep-crawl extraction
• config/sources.yaml — Per-source rate limit config

What This Epic Adds

1. DMC Finder crawler — New source with 10,000+ DMCs across 120+ countries
2. ADMEI enrichment — Enrich existing records with ADMEI member data
3. Cross-source deduplication — Merge records across all sources into canonical entities

---

Acceptance Criteria

1. DMC Finder Crawler

• ☐ Create crawlers/dmc/dmc_finder.py extending BaseCrawler
• ☐ Implement fetch_list() — paginate DMC Finder directory (10,000+ entries)
• ☐ Implement parse_list() — extract DMC name, country, services, specialties, contact info
• ☐ Implement fetch_detail() — fetch individual DMC profile pages
• ☐ Implement parse_detail() — extract full contact details (email, phone, website, social)
• ☐ Handle pagination (offset-based or cursor-based)
• ☐ Handle rate limiting (respect robots.txt, configurable RPS)
• ☐ Handle connection errors with exponential backoff
• ☐ Map extracted fields to DMC model schema
• ☐ Write unit tests for parser (mock HTML responses)
• ☐ Write integration tests (live crawl with rate limiting)

2. ADMEI Enrichment

• ☐ Create crawlers/dmc/admei.py — ADMEI member directory crawler
• ☐ Implement member list extraction (company name, member type, website)
• ☐ Implement member profile enrichment (contact details, certifications, destinations)
• ☐ Build enrichment pipeline: match ADMEI members to existing DMC records by name/website
• ☐ Merge ADMEI data into existing records (don't overwrite existing fields, fill gaps)
• ☐ Mark enriched records with admei_member: true flag
• ☐ Write tests for matching logic and merge behavior

3. Cross-Source Deduplication

• ☐ Create intelligence/dedup/dmc_deduplicator.py
• ☐ Implement entity matching by:
  • Company name (fuzzy match with normalization: lowercase, strip punctuation, handle "LLC"/"Ltd"/"Inc")
  • Website URL (normalize: strip protocol, www., trailing slash)
  • Email domain match
  • Phone number match (after E.164 normalization)
  • Location proximity (same city + country)
• ☐ Implement merge strategy:
  • Keep the record with highest confidence as the base
  • Fill missing fields from lower-confidence matches
  • Preserve all unique source URLs
  • Merge unique specialties, destinations, services
  • Keep the most recent updated_at
• ☐ Build dedup pipeline:
  1. Group candidates by country (reduce search space)
  2. Pairwise comparison within groups
  3. Score similarity (weighted combination of name, website, email, phone, location)
  4. Merge above threshold (configurable, default 0.85)
  5. Log all merge decisions for audit
• ☐ Write unit tests for matching functions
• ☐ Write integration tests for full dedup pipeline
• ☐ Build CLI command: intelligence dedup dmc --dry-run to preview merges

4. Crawler Orchestration

• ☐ Add DMC Finder to config/sources.yaml with rate limits
• ☐ Add ADMEI to config/sources.yaml with rate limits
• ☐ Add intelligence crawl dmc-finder CLI command
• ☐ Add intelligence crawl admei CLI command
• ☐ Add intelligence enrich dmc CLI command (runs ADMEI matching)
• ☐ Add intelligence dedup dmc CLI command
• ☐ Build full DMC pipeline command: intelligence pipeline dmc (crawl → enrich → dedup)
• ☐ Add progress logging (records crawled, matched, merged, errors)

5. Storage Extensions

• ☐ Add source_urls field to DMC model (list of all source URLs for traceability)
• ☐ Add admei_member boolean field to DMC model
• ☐ Add dedup_group_id field to track merged records
• ☐ Add crawled_at timestamp per source (for incremental crawl support)
• ☐ Write migration for schema changes
• ☐ Update SQLite store for new fields

6. Tests

• ☐ Unit tests for DMC Finder parser
• ☐ Unit tests for ADMEI parser
• ☐ Unit tests for dedup matching functions
• ☐ Unit tests for merge strategy
• ☐ Integration tests: crawl DMC Finder → store → dedup
• ☐ Integration tests: enrich with ADMEI → merge
• ☐ Edge cases: duplicate within same source, cross-source conflicts, missing fields

---

File Structure (created by this brief)

marchay-intelligence/
├── intelligence/
│   ├── crawlers/
│   │   └── dmc/
│   │       ├── __init__.py
│   │       ├── inside_travel.py      (existing)
│   │       ├── traveller_made.py     (existing)
│   │       ├── dmc_travel.py         (existing)
│   │       ├── dmc_finder.py         ← NEW
│   │       └── admei.py              ← NEW
│   ├── dedup/
│   │   ├── __init__.py               ← NEW
│   │   └── dmc_deduplicator.py       ← NEW
│   ├── models/
│   │   └── dmc.py                    (modify: add source_urls, admei_member, dedup_group_id, crawled_at)
│   ├── storage/
│   │   └── sqlite_store.py           (modify: handle new fields)
│   └── cli.py                        (modify: add crawl/enrich/dedup commands)
├── config/
│   └── sources.yaml                  (modify: add dmc_finder, admei)
├── tests/
│   ├── test_crawlers/
│   │   ├── test_dmc_finder.py        ← NEW
│   │   └── test_admei.py             ← NEW
│   ├── test_dedup/
│   │   └── test_dmc_deduplicator.py  ← NEW
│   └── test_models/
│       └── test_dmc.py               (modify: test new fields)
└── migrations/
    └── versions/
        └── xxx_add_dmc_enrichment_fields.py  ← NEW

---

Implementation Phases

Phase 1: DMC Finder Crawler (Days 1-3)

1. Research DMC Finder directory structure (URL patterns, pagination, data format)
2. Create dmc_finder.py with BaseCrawler inheritance
3. Implement fetch_list() and parse_list()
4. Implement fetch_detail() and parse_detail()
5. Add to config/sources.yaml
6. Write unit tests with mock HTML
7. Test live crawl with rate limiting

Phase 2: ADMEI Enrichment (Days 3-5)

1. Research ADMEI member directory structure
2. Create admei.py crawler
3. Implement member list extraction
4. Implement profile enrichment
5. Build matching pipeline (name/website normalization)
6. Build merge logic (fill gaps, don't overwrite)
7. Write tests

Phase 3: Cross-Source Deduplication (Days 5-8)

1. Create dedup/dmc_deduplicator.py
2. Implement name normalization and fuzzy matching
3. Implement website URL normalization
4. Implement email/phone matching
5. Build similarity scoring function
6. Build merge strategy
7. Add CLI commands with --dry-run support
8. Write unit and integration tests

Phase 4: Storage & Schema (Days 8-9)

1. Add new fields to DMC model
2. Write Alembic migration
3. Update SQLite store
4. Update schema registry
5. Test migration up/down

Phase 5: CLI & Pipeline (Days 9-10)

1. Add intelligence crawl dmc-finder command
2. Add intelligence crawl admei command
3. Add intelligence enrich dmc command
4. Add intelligence dedup dmc command
5. Build intelligence pipeline dmc orchestrator
6. Add progress logging and error reporting

Phase 6: Integration Testing (Days 10-12)

1. Full pipeline test: crawl → store → enrich → dedup
2. Verify dedup quality (sample manual review)
3. Performance test with full dataset
4. Error handling and recovery testing

Phase 7: Polish (Day 12)

1. Add docstrings and type hints
2. Run all tests (ruff, mypy)
3. Update README.md
4. Write build summary

---

Dependencies

Dependency	Version	Purpose
httpx	^0.27.0	Async HTTP client (existing)
beautifulsoup4	^4.12.0	HTML parsing (existing)
rapidfuzz	^13.0.0	Fuzzy string matching for dedup
python-Levenshtein	^0.25.0	Fast string distance (rapidfuzz optional dep)
pydantic	^2.7.0	Data models (existing)
aiosqlite	^0.20.0	Async SQLite (existing)
typer	^0.12.0	CLI framework (existing)
pytest	^8.2.0	Testing (existing)
pytest-asyncio	^0.23.0	Async test support (existing)

---

Data Volume Estimates

Source	Records	Crawl Time (est.)
DMC Finder	10,000+	~2-3 hours (at 2 RPS)
ADMEI	~200-500	~10-15 minutes
Inside.travel	~192	~5 minutes (existing)
Traveller Made	~39	~2 minutes (existing)
Total raw	~10,500+
After dedup	~8,000-10,000

---

Success Metrics

Metric	Target
DMC records indexed	10,000+
Country coverage	120+ countries
Email coverage	>90%
Phone coverage	>75%
Dedup precision	>95% (manual sample review)
Dedup recall	>90% (catches most duplicates)
False positive rate	<5% (merges that shouldn't happen)
Crawl completion rate	>99% (with retries)