Divider

Gather interesting urls with extensions from a given file.

What this will do

This will process files generated by tools like waybackurls,gau,etc containing urls and grab only urls ending with file extensions.

Installation

Clone the repository:
        git clone https://github.com/Aspasht/Divider.git

Install dependencies:
        pip install -r Requirements.txt

Required tools assigned to fetch flag:
        chmod +x install.sh
        ./install.sh
        Or you can install it manually by using following commands:
        go install github.com/lc/gau/v2/cmd/gau@latest
        go install github.com/tomnomnom/waybackurls@latest

Available Commands

  $ python divider.py -f myurlfile.txt
  $ python divider.py --fetch hosts.txt --req

Example

$ cat myUrls.txt
    https://example.com/test/test3
    https://example.com/test.css
    https://example.com/robots.txt
    https://example.com/sitemap.xml
    https://example.com/test?q=testvalue

$ python divider.py -f myUrls.txt
    https://example.com/robots.txt
    https://example.com/sitemap.xml

Usage

usage: divider.py [-h] -f FILE [-req REQUEST]
options:
  -h, --help                             show this help message and exit
  -f FILE, --file FILE                   Please add target file as argument!
  -o OUTPUT, --output OUTPU              Save output to a file!
  --req REQUEST, --request REQUEST       Send request for previously generated urls! (Default=False)
  --greq GREQUEST, --grequest GREQUEST   Send request for previously generated urls using grequest! (Default=False)
  --fetch FETCH, --fetch FETCH           Fetch urls using tools like waybackurls and gau {must be preinstalled}!

Commands

$ python divider.py -f myurls.txt # Grab urls with extensions without sending any requests.
$ python divider.py -f myurls.txt --req # Send normal requests using httpx module.
$ python divider.py -f myurls.txt --greq # Send requests using grequests module.
    Note: Use grequests flag if your resulting urls are less in number!
$ python divider.py --fetch hosts.txt --req -o out.txt # Save output to a file.