Single-threaded Web Crawler

(Front-end part - A compatible Android app is available at Career_Crawler)

This projects includes a single-threaded web crawler. It is designed to crawl sites with content in English and Hindi.

Features

Single-threaded
Follows breadth-first strategy
Handles various MIME types
Can overcome anti-crawling traps deployed by web administrators - looks humaly
Can crawl sites that are in UTF-8 format, particularly in English and Hindi languages
Can normalise relative paths written in different styles
Detects broken/missing links and handles a variety of HTTP errors
Logo extractor to download logos of institutions
Can be easily integrated with Selenium to see the live crawling!!

We crawled educational sites situated near-by. Some of them have been plotted on the map:

Name		Name	Last commit message	Last commit date
Latest commit History 84 Commits
img		img
.gitignore		.gitignore
Crawler.py		Crawler.py
Data.py		Data.py
Institutes_Added_to_Check_via_Program (copy)		Institutes_Added_to_Check_via_Program (copy)
README.md		README.md
crawl_and_find.py		crawl_and_find.py
crawl_on_demand.py		crawl_on_demand.py
db_connector.py		db_connector.py
list		list
modules.py		modules.py
scrape_google_maps.py		scrape_google_maps.py
scrape_logos.py		scrape_logos.py
url_utilities.py		url_utilities.py