Scrapengine

A generic scraping engine that schedules, loads, runs crawlers and outputs to an external data store.

Objectives

• Self contained pluggable scrapers
• Scraper execution engine that loads and executes the scrapers at pre-configured intervals
• Emit output to pre-configured data stores
• Web dashboard: admin, monitoring, and metrics

Status

Pre-development

Documentation