Home

Jump to bottom Edit New page

Ayush Agarwal edited this page Dec 20, 2017 · 2 revisions

Welcome to the WIKI!

Features of the project:

Web GUI for viewing crawl inventory.
Automated Pipelines
- Automatic Spider Id Assignment
- Automatic Duplicates Handling
- Automatic Data/Date Parsing
- Automatic Database Management
- Singleton Database Conection Implementation
Automatic Scheduling (Requires Scrapyd)
Scrapyd Compatible
Database Connection Management

www.thisisayush.com

Add a custom sidebar

Clone this wiki locally