Skip to content
This repository has been archived by the owner on Jul 8, 2018. It is now read-only.

Releases: crawlerclub/x

Release v0.1.0

20 Jul 02:32
Compare
Choose a tag to compare
Release v0.1.0 Pre-release
Pre-release

Features

  • Focused crawler based on configuration of xpath rules.
  • Crawler configuration editor, a web based json editor.
  • Concurrency crawling by goroutines.
  • Multiple crawlers support, each crawler has its own priority.
  • Crawled items could be processed further by arbitrary javascript codes.
  • Crawled items could be saved to configured ElasticSearch server automatically.
  • Support incremental crawling.
  • Dashboard for monitoring system loads.