crab

A versatile tool to crawl dozens of URLs from a given source, like a sitemap or an URL list.

Useful for:

• Warming site caches
• Checking response times
• Identifying dead or broken pages

Install

Binaries

You can download the newest release from here for Linux (including .deb and .rpm), macOS and Windows.

Build Command (Personal & Development Usage)

#Linux (Debian/Ubuntu) & MacOS
$ go build -o crab cmd/crab/main.go

#Windows
$ go build -o crab.exe cmd/crab/main.go

Docker

Docker Hub

$ docker pull atomicptr/crab
# Example
$ docker run --rm atomicptr/crab --help
$ docker run --rm atomicptr/crab crawl:sitemap https://domain.com/sitemap.xml

Arch Linux

AUR

Nix

Not available in nixpkgs but I have my own nix repository which you can use:

let
  atomicptr = import (fetchTarball "https://github.com/atomicptr/nix/archive/refs/heads/master.tar.gz") {};
in
{
  environment.systemPackages = with pkgs; [
    atomicptr.crab
  ];
}

Homebrew

$ brew install atomictr/tools/crab

Scoop

$ scoop bucket add atomicptr https://github.com/atomicptr/scoop-bucket
$ scoop install crab

Usage

Crawl singular URLs:

$ crab crawl https://domain.com https://domain.com/test
{"status": 200, "url": "https://domain.com", ...}
...

Crawl through a sitemap:

$ crab crawl:sitemap https://domain.com/sitemap.xml

Replace all URLs with a different one:

$ crab crawl:sitemap https://domain.com/sitemap.xml --prefix-url=https://staging.domain.com

Add some cookies/headers:

$ crab crawl:sitemap https://domain.com/sitemap.xml --cookie auth_token=12345 --header X-Bypass-Cache=1

Filter by Status Code

You can filter the output by it's status code

# This will only return responses with a 200 OK
$ crab crawl:sitemap https://domain.com/sitemap.xml --filter-status=200
# This will only return responses that are not OK
$ crab crawl:sitemap https://domain.com/sitemap.xml --filter-status=!200
# This will only return responses between 500-599 (range)
$ crab crawl:sitemap https://domain.com/sitemap.xml --filter-status=500-599
# This will only return responses with 200 or 404 (multiple, be aware if one condition is true they all are)
$ crab crawl:sitemap https://domain.com/sitemap.xml --filter-status=200,404
# This will only return responses with a code greater than 500
$ crab crawl:sitemap https://domain.com/sitemap.xml --filter-status=>500

Save Url List in File

You can save the url list to a file

# This will save the output to a file called output.txt
$ crab crawl:sitemap https://domain.com/sitemap.xml --output-file ./output/output.txt

Save Output to JSON

You can save the output to a JSON file

# This will save the output to a file called output.json
$ crab crawl:sitemap https://domain.com/sitemap.xml --output-json ./output/output.json

License

MIT