README.md

Crawler4U: A general purpose focused crawler

Overview

Crawler4U is a general purpose focused crawling and scraping tool based on json format configurations.

Install

The Golang way:

go get crawler.club/crawler

Or download the pre-built binaries from releases for you system.

Build from source

Before you can build the crawler from source. A workable golang development environment is needed. Downlad golang here and then execute the following commands.

go get -d crawler.club/crawler
cd $GOPATH/src/crawler.club/crawler
make

Usage

中文

Companies using crawler.club/crawler

• Huawei
• Baidu
• Toutiao
• Zenia