SubScraper

     
Overview    🔹    Usage    🔹    Contribute

SubScraper is a subdomain enumeration tool that uses a variety of techniques to find subdomains of a given target. Subdomain enumeration is especially helpful during penetration testing and bug bounty hunting to uncover an organization's attack surface.

Depending on the CMD arguments applied, SubScraper can resolve DNS names, request HTTP(S) information, and perform CNAME lookups for takeover opportunities during the enumeration process. This can help identify next steps and discover patterns for exploitation.

Key Features

• Modular design makes it easy to add new techniques/sources.
• Various levels of enumeration for additional data gathering.
• Allows for multiple target inputs, reading from .txt or STDIN.
• Windows CLI compatibility.
• Generate output files in .txt or .csv format.

Installation

Python

The following can be used to install SubScraper on Windows, Linux, & MacOS:

git clone https://github.com/m8sec/subscraper
cd subscraper
pip3 install -r requirements.txt

Poetry

Install and run SubScraper using Poetry:

git clone https://github.com/m8sec/subscraper
cd subscraper
poetry install
poetry run subscraper -h

Docker

You can build a docker image and run subscraper from Docker:

git clone https://github.com/m8sec/subscraper
cd subscraper
docker build -t m8sec/subscraper .

# Display help
docker run --rm m8sec/subscraper

# Example scanning a site
docker run --rm m8sec/subscraper -d example.com

Updates:

Use the configuration file at ~/.config/subscraper/config.json to store API keys for easy reuse.

If updating to a newer version after v4.0.0, use the -update argument to pull a new copy of the config file and ensure compatibility - Note: This will remove any existing key entries.

Modules

A full list of modules can be found using the -ls command line argument:

bevigil              - BeVigil OSINT API for scraping mobile application for subdomains (API Key Req)
crt.sh               - Subdomains enumeration using cert.sh.
virustotal           - Lookup subdomain on VirusTotal (API Key Req)
dnsrepo              - Parse dnsrepo.noc.org without an API key - 150 result limit
chaos                - Project Discovery's Chaos (API Key Req)
certspotter          - Use Certspotter API to collect subdomains
bufferover           - Query Bufferover.run API (API Key Req)
alienvault           - Find subdomains using AlienVault OTX
redhuntlabs          - RedHunt Labs recon API (API Key Req)
archive              - Use archive.org to find subdomains.
dnsdumpster          - Use DNS dumpster to enumerate subdomains.
censys.io            - Gather subdomains through censys.io SSL cert Lookups. (API Key Req)
shodan               - Get subdomains with Shodan (API Key Req)

Usage

Command Line Args

SubScraper Options:
  -debug                Enable debug logging
  -update               Update config file (Will remove existing entries)
  -config CONFIG        Override default config location
  -silent               Show subdomains only in output
  -threads THREADS, -T THREADS    Max threads for enumeration (65*).
  -t TIMEOUT                      set connection timeouts (3*)
  -d DOMAIN, --domain DOMAIN      Target domain input (domain, .txt, STDIN, etc.

Module Options:
  -ls                   List SubScraper enumeration modules.
  -m MODULES            Execute module(s) by name or group (all*).
  -module-only          Execute modules only not brute force

Bruteforce Options:
  -w WORDLIST           Custom wordlist for DNS brute force.
  -ns NS                Comma separated nameservers to use

Enumeration Options:
  -r, -resolve          Resolve IP address for each subdomain identified.
  -c, -cname            Perform CNAME lookup for subdomain takeover checks
  -http                 Probe for active HTTP services.
  -http-port HTTP_PORT  HTTP ports to check, comma separated (80,443*)

Output Options:
  -nc, -no-color        Disable color output
  -active               Only report active subdomains with resolved IP
  -csv                  Create CSV output report
  -o REPORT             Output file

Example Inputs

python3 subscraper.py -d example.com -resolve -http -module-only
python3 subscraper.py -d example.com -cname -m none -o sub_report.csv -csv
cat domains.txt | python3 subscraper.py -active -silent

Contribute

Contribute to the project by:

• Like and share the tool!
• Create an issue to report new enumeration techniques
• OR, better yet, develop a module and initiate a PR.