Usage Configure urls to parse in config.py. Then, run: scrapy runspider broken_links_spider.py -o output.json Then check 404 items in the output.json file.