DMC Data Enrichment - Build Summary

Date: August 21, 2026 Status: Complete Epic: DMC Data Enrichment

---

What Was Built

Crawlers

Crawler	Source	Method	Status
DMCFinderCrawler	DMC Finder (41,000+ DMCs)	Playwright + Load More	Working
ADMEICrawler	ADMEI Members	Playwright + Pagination	Working

BaseCrawler (intelligence/crawlers/base.py)

• Rate limiting (configurable RPS)
• Retry with exponential backoff
• Concurrency control via semaphores
• HTTP client (httpx) for standard pages
• Playwright integration for JavaScript-rendered sites
  • Stealth mode (realistic user agent, viewport)
  • fetch_with_playwright() method
  • Context management (browser, pages)

DMCFinderCrawler (intelligence/crawlers/dmc/dmc_finder.py)

• Uses Playwright (site loads via JavaScript)
• Clicks #btnToLoadMorePost to load more results
• Parses .member_results containers
• Extracts: name, detail_url, location, phone
• Base URL: https://dmcdirectory.dmcfinder.com/search_results

ADMEICrawler (intelligence/crawlers/dmc/admei.py)

• Uses Playwright (Knockout.js rendering)
• Page-based pagination: ?pageNumber=1, ?pageNumber=2
• Parses #members-container .member elements
• Extracts: name, address, detail_url
• Base URL: https://www.admei.org/find-a-dmc

---

CLI Commands

# Crawl DMC Finder (1 page = ~24 DMCs)
uv run python -m intelligence.cli crawl dmc-finder --pages 1

# Crawl ADMEI (1 page = ~12 members)
uv run python -m intelligence.cli crawl admei --pages 1

# Full pipeline
uv run python -m intelligence.cli pipeline dmc

# Deduplication
uv run python -m intelligence.cli dedup dmc --threshold 0.85
uv run python -m intelligence.cli dedup dmc --dry-run

---

Technical Decisions

Why Playwright?

Both DMC Finder and ADMEI use JavaScript to render content:

• DMC Finder: Load More button (#btnToLoadMorePost)
• ADMEI: Knockout.js data binding

Standard HTTP requests return empty containers.

Stealth Configuration

ADMEI has a WAF that blocks headless browsers. Bypassed with:

browser = await playwright.chromium.launch(
    headless=True,
    args=["--disable-blink-features=AutomationControlled"],
)
context = await browser.new_context(
    user_agent="Mozilla/5.0 ...",
    viewport={"width": 1920, "height": 1080},
    locale="en-US",
)

---

Test Results

70 passed in 0.27s
ruff: All checks passed
mypy: No issues found in 26 source files

---

Data Volume

Source	Per Page	Notes
DMC Finder	~24 DMCs	Load More pattern
ADMEI	12 members	Standard pagination

---

Acceptance Criteria

• [x] DMC Finder crawler with Playwright
• [x] ADMEI crawler with Playwright
• [x] Stealth configuration for WAF bypass
• [x] Standardized --pages flag
• [x] Unit tests for parsers
• [x] Integration tests
• [x] CLI commands
• [x] Deduplication with fuzzy matching

---

Next Steps

• [ ] Store crawled data to SQLite
• [ ] Run full pipeline (crawl → store → dedup)
• [ ] Add progress logging
• [ ] Performance testing with multiple pages