DMC Vertical

Complete the DMC vertical from spike to production.

LOE: ~50-70 hours

---

Epic: DMC Data Enrichment

Plan: Expand the DMC dataset from the initial spike by integrating DMC Finder (10,000+ DMCs, 120+ countries), Traveller Made, and the Virtuoso advisor network (20,000+ luxury advisors).

Architectural Context: The initial spike validated DMC data extraction. This epic scales to production volume by adding additional sources and deduplicating across them. Cross-source deduplication ensures a single canonical DMC record per company.

Tasks

• Implement DMC Finder crawler (10,000+ DMCs, 120+ countries)
• Build Traveller Made enrichment
• Implement Virtuoso advisor network (20,000+ luxury advisors)
• Build cross-source deduplication

---

Epic: DMC Data Quality

Plan: Validate and normalize extracted DMC data: email validation via MX records, phone normalization to E.164, confidence scoring, and department-specific email extraction.

Architectural Context: Raw extraction produces inconsistent data. Email validation ensures deliverability. Phone normalization enables consistent matching. Confidence scoring flags low-quality extractions for manual review. Department-specific emails (reservations@, sales@, booking@) improve routing accuracy.

Tasks

• Implement email validation (MX record check)
• Build phone normalization (E.164 format)
• Implement confidence scoring for extracted values
• Build department-specific email extraction (reservations@, sales@, booking@)

---

Epic: DMC Advanced Extraction

Plan: Handle complex extraction scenarios: JavaScript-rendered contact pages, PDF brochures, and cookie consent walls that block content.

Architectural Context: Many DMC websites use JavaScript rendering, PDF brochures for contact info, and cookie consent modals that block page content. Playwright handles JS rendering; PDF parsing extracts contact details; cookie consent handling bypasses blocks.

Tasks

• Implement JavaScript-rendered contact page handling
• Build PDF brochure parsing
• Implement cookie consent handling for sites that block content

---

Epic: DMC Production Operations

Plan: Containerize the DMC crawler, implement incremental crawl support, enforce rate limits per source, and add error alerting.

Architectural Context: Production operations ensure the DMC crawler runs reliably at scale. Docker containerization enables deployment. Incremental crawling reduces redundant work. Rate limiting prevents source blocks. Error alerting enables proactive issue resolution.

Tasks

• Docker containerization
• Implement incremental crawl support
• Build rate limit enforcement per source
• Implement error alerting