Skip to content

Commit

Permalink
升级正式版本:v1.1.0
Browse files Browse the repository at this point in the history
  • Loading branch information
xuxueli committed Nov 8, 2017
1 parent 3741295 commit 2bc76de
Show file tree
Hide file tree
Showing 4 changed files with 6 additions and 7 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ XXL-CRAWLER 是一个灵活高效、面向对象的分布式爬虫框架。一
- 10、主动停顿:爬虫线程处理完页面之后进行主动停顿,避免过于频繁被拦截;
- 11、单个页面支持抽取一个或多个PageVO;
- 12、代理IP:对抗反采集策略规则WAF;
- 13、动态代理:支持运行时动态调整代理池,甚至自定义代理池路由策略
- 13、动态代理:支持运行时动态调整代理池,以及自定义代理池路由策略

## Communication

Expand Down
8 changes: 4 additions & 4 deletions doc/XXL-CRAWLER官方文档.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ XXL-CRAWLER 是一个灵活高效、面向对象的分布式爬虫框架。一
- 10、主动停顿:爬虫线程处理完页面之后进行主动停顿,避免过于频繁被拦截;
- 11、单个页面支持抽取一个或多个PageVO;
- 12、代理IP:对抗反采集策略规则WAF;
- 13、动态代理:支持运行时动态调整代理池,甚至自定义代理池路由策略
- 13、动态代理:支持运行时动态调整代理池,以及自定义代理池路由策略

### 1.4 下载

Expand Down Expand Up @@ -198,11 +198,11 @@ ProxyMaker(代理生成器):实现代理支持的组件。支持设置代
- 3、修复PageVO注解失效的问题;
- 4、属性注解参数attributeKey调整为selectVal;
- 5、代理IP:对抗反采集策略规则WAF;
- 6、动态代理:支持运行时动态调整代理池,甚至自定义代理池路由策略
- 6、动态代理:支持运行时动态调整代理池,以及自定义代理池路由策略

### TODO LIST
- 1、爬虫超时重试
- 2、扩展SelectType,比如HasClass等
- 1、扩展SelectType
- 2、失败重试
- 3、动态UserAgent;
- 4、js渲染;

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.xuxueli</groupId>
<artifactId>xxl-crawler</artifactId>
<version>1.0.1-SNAPSHOT</version>
<version>1.1.0</version>
<packaging>jar</packaging>

<name>${project.artifactId}</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,6 @@ public void parse(Document html, PageVo pageVo) {
if (proxyPool!=null && proxyPool.size()>0) {
for (PageVo pageVo: proxyPool) {
try {
//
Document html = JsoupUtil.load("http://2017.ip138.com/ic.asp",
null,
null,
Expand Down

0 comments on commit 2bc76de

Please sign in to comment.