Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

google map scraper using python playwright

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

what is the best web scraping API service? Research through benchmarks

A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.

Another curated list of Python frameworks

JAW: A Graph-based Security Analysis Framework for Client-side JavaScript

Sistema de Mapeamento de Conflitos (MapCon)

Library for Rapid (Web) Crawler and Scraper Development

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Collection of practical codes for Savitribai Phule Pune University's Information Retrieval Lab (410247) .

It documents my journey of learning and building web scrapers from the ground up using Python, based on the book "Web Scraping with Python" by Mitchell Ryan. It includes code examples, projects, and notes as I explore techniques for collecting data from the modern web.

This repository contains a simple web crawler program.

Example site for web scraping tutorials

An easy-to-use library for the SpeedyShot Capture service.

Framework Docs AI is a powerful SaaS solution for managing framework documentation. It automatically scrapes documentation, builds a comprehensive knowledge base, and uses advanced language models to provide accurate responses to user queries. Enhance productivity and streamline your documentation process with Framework Docs AI.

MetadataHarvester is an advanced file metadata extraction tool designed for cybersecurity professionals, researchers, and analysts.

The All in One Framework to build Awesome Scrapers.