This is a "web" crawler. The main feature of this repo is the multi threading implementation for web crawling in python.
This only contains the code for the crawler and not the simulator for the web. TThe requirements.txt give the libraries required.
The crawler class contains the functions responsible for thread pools and the crawling function itself. The constants file under config gives us some parameters to setup the crawler.