Single domain web crawler

Crawl a single domain outputting the website's sitemap as a list of pages with each page's static assets and links to other pages.

Running it

1. Install the dependencies from requirements.txt:

$ pip install -r requirements.txt

3. And then run crawl.py:

$ python crawl.py