Extractors

Extractors turn raw HTML/JSON into structured data. The platform has two shared extractors used by every vertical.

Contact Extractor

extractors/contact.py extracts contact information from any HTML page. It uses regex patterns tuned for common formats.

What It Extracts

• Emails — RFC-5322-ish regex, filtered to exclude common false positives (images, example.com, etc.)
• Phones — international format, with country code detection
• Social links — Instagram, Facebook, LinkedIn, Twitter/X handles
• WhatsApp — wa.me links and phone numbers
• Physical addresses — street + city + postal code + country patterns
• Department-specific emails — reservations@, sales@, booking@ (DMC TODO)

How It Works

The contact extractor takes a list of HTML pages (often from the deep-crawl, see below) and merges the extracted data into a single Contact object.

from extractors.contact import extract_contact

contact = extract_contact([
    html_from_homepage,
    html_from_contact_page,
    html_from_about_page,
])
# contact.emails, contact.phones, contact.social, contact.address

The DMC spike validated this against hundreds of heterogeneous business sites with 95% email coverage and 80% phone coverage.

Website Deep-Crawler

extractors/website.py is the deep-crawl pattern. Given a business's website URL, it:

1. Fetches the homepage
2. Looks for links to contact, about, team, and similar pages
3. Fetches those pages
4. Returns the union of HTML for the contact extractor to process

How It Works

from extractors.website import deep_crawl

pages = deep_crawl("https://example-dmc.com")
# pages = [homepage_html, contact_html, about_html, ...]

from extractors.contact import extract_contact
contact = extract_contact(pages)

This is what produces the 95% email / 80% phone coverage for DMCs. Without deep-crawling, the coverage would be much lower.

Limitations

• JavaScript-rendered pages — currently a TODO. Many modern sites render contact info client-side; BeautifulSoup sees an empty page.
• PDF brochures — currently a TODO. Some DMCs publish contact info only in PDF brochures.
• Cookie consent walls — currently a TODO. Some sites block content until cookies are accepted.

Adding a New Extractor

If a vertical needs extraction that doesn't fit the contact or website patterns, add a new extractor:

# extractors/menu.py (hypothetical, for dining)
def extract_menu(html: str) -> Menu:
    ...

The pattern is: pure function in, Pydantic model out.

Quality Signals

The extractors tag every extracted value with a confidence signal:

• High — matches a known pattern on a dedicated contact page
• Medium — matches a pattern on a general page
• Low — partial match, ambiguous format, or extracted from navigation/footer

Confidence scores feed into the data quality metrics. See Observability.