A web crawler and scraper for Rust

An advanced web crawler powered by large language models, featuring adaptive rate limiting, content deduplication, dynamic content extraction, continuous learning, proxy management, and intelligent prioritization of links. It also respects robots.txt rules and parses sitemaps for efficient crawling

Web-QueryAI