Crawlers

Crawlers fetch raw data from sources. Every source-specific crawler extends BaseCrawler. The base provides the cross-cutting concerns (rate limiting, retry, concurrency); the subclass implements the source-specific fetching and parsing.

BaseCrawler

crawlers/base.py is the abstract base for all source crawlers. It provides:

• Rate limiting — configurable requests per second per source
• Retry logic — exponential backoff with jitter, bounded attempts
• Concurrency control — semaphore-bounded parallel requests
• HTTP session management — connection pooling, default headers, timeout
• Logging — structured logs per request with source, URL, status, latency
• Error categorization — transient, permanent, rate-limited

Every concrete crawler implements:

• fetch_list() — get the source's directory/listing page
• parse_list(html) — extract entries from a listing page
• fetch_detail(entry) — fetch a single entity's detail page (if needed)
• parse_detail(html) — extract structured data from a detail page
• source_name — a string identifier for the source

Rate Limiting

Per-source rate limits are declarative, configured in config/sources.yaml:

sources:
  inside_travel:
    rate_limit_per_second: 2
    max_concurrent: 5
    retry_max_attempts: 3
    timeout_seconds: 30
  traveller_made:
    rate_limit_per_second: 1
    max_concurrent: 3
    retry_max_attempts: 3
    timeout_seconds: 30

The BaseCrawler reads this config and enforces it. No hardcoded limits in crawler code.

DMC Crawlers (Spike)

Crawler	Source	Status
inside_travel.py	Inside.travel directory	Works (~192 DMCs)
traveller_made.py	Traveller Made members	Works (~39 DMCs)
dmc_travel.py	DMC.travel	Parsing issues — needs work

How a DMC Crawl Works

1. fetch_list() — paginate the directory, yield entries with name and detail URL
2. For each entry, optionally fetch_detail() and parse_detail() for richer data
3. Pass the parsed record to the contact extractor (see Extractors)
4. Upsert into SQLite via the storage layer (see Storage)

Adding a New Source

# crawlers/dining/resy.py
from crawlers.base import BaseCrawler

class ResyCrawler(BaseCrawler):
    source_name = "resy"

    def fetch_list(self, params):
        # Return raw HTML/JSON from Resy's directory
        ...

    def parse_list(self, response):
        # Yield raw entries
        ...

    def fetch_detail(self, entry):
        # Optional: fetch a detail page
        ...

    def parse_detail(self, response):
        # Optional: extract structured data
        ...

That's it. Rate limiting, retry, concurrency, and logging are inherited.

Anti-Bot Considerations

Sources may block aggressive crawlers. Mitigations:

• Polite defaults — the rate limit defaults are conservative
• User agent rotation — a small pool of realistic user agents
• Cookie acceptance — handle cookie consent popups where they appear (DMC TODO)
• Backoff on 429/403 — exponential backoff triggers on rate-limit responses
• Per-source health tracking — if a source starts blocking, alert and pause

We do not use headless browsers, residential proxies, or CAPTCHA solvers. If a source requires those, we treat it as a partner conversation, not a technical challenge to defeat.

Incremental Crawling (Planned)

For verticals where data changes slowly (DMCs, hotels), full re-crawls are wasteful. The platform supports incremental crawls:

• Re-crawl only records with crawled_at older than N days
• Re-crawl only sources whose directory page has changed
• Re-crawl on demand (e.g., when an agent flags a stale record)

The DMC spike crawls everything every time. Incremental is a post-spike enhancement (see Verticals).