CLI & Orchestration

The platform ships with a Click-based CLI for ad-hoc operation, scheduled runs, and orchestration.

Commands

# Crawl a single source, limited to N records (testing)
uv run python crawler.py crawl --source inside_travel -n 5 --no-deep-crawl

# Full crawl across all configured sources
uv run python crawler.py crawl --source all

# Search
uv run python crawler.py search --destination "Greece"
uv run python crawler.py search --service "Luxury Travel" -n 20

# Stats
uv run python crawler.py stats

# Export to JSON
uv run python crawler.py export -o output/dmc_data.json

# Serve the MCP server (stdio)
uv run python crawler.py serve

Orchestration

Scheduled Crawls

Each vertical has a schedule. DMCs refresh weekly; hotels refresh daily; flights refresh hourly. The schedule lives in a config file:

# config/schedule.yaml
verticals:
  dmc:
    full_crawl: "0 3 * * 0"      # Sundays at 3am
    incremental: "0 3 * * 1-6"  # Daily except Sunday
  dining:
    full_crawl: "0 4 * * 1"      # Mondays at 4am
  hotels:
    full_crawl: "0 5 * * *"      # Daily at 5am

The scheduler runs as a separate process (cron, k8s CronJob, or a dedicated scheduler service). It invokes the CLI for each scheduled run.

Run State

Every crawl run writes a state record:

{
  "run_id": "uuid",
  "vertical": "dmc",
  "source": "inside_travel",
  "started_at": "...",
  "finished_at": "...",
  "status": "success" | "partial" | "failed",
  "records_processed": 192,
  "records_added": 12,
  "records_updated": 180,
  "errors": []
}

State is written to a runs table in SQLite. The platform uses it to detect stuck runs, calculate freshness, and report per-source health.

Incremental Crawls (Planned)

Full re-crawls are wasteful for slow-changing data. The scheduler supports incremental mode:

uv run python crawler.py crawl --source inside_travel --incremental --max-age-days 30

Incremental crawls skip records whose crawled_at is newer than the threshold, unless the directory page signals a change.

Operational Runbook

Scenario	Action
Source is rate-limiting	Reduce rate limit, alert, check if our IP got flagged
Source structure changed	Parser breaks, alert fires, manual update to crawler
Crawl is slow	Check concurrency settings, check source health
Data quality regressed	Check coverage metrics, investigate extractor changes
Disk is filling	Check backup retention, archive old data
MCP server won't start	Check SQLite file integrity, check port availability