Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

爬取不到数据,启动 work 页面 输出的都是一些爬取失败 和 warning 信息 类似: #211

Open
gaoshangle opened this issue Aug 23, 2020 · 1 comment

Comments

@gaoshangle
Copy link

在提交Issue之前请先回答下面问题,谢谢!

1.你是怎么操作的?

尽量把你的操作过程描述清楚,最好能够复现问题。

2.你期望的结果是什么?

3.实际上你得到的结果是什么?

4.你使用的是哪个版本的WeiboSpider? 你的操作系统是什么?是否有读本项目的常见问题

@gaoshangle
Copy link
Author

gaoshangle commented Aug 23, 2020

1.按照配置进行,之前爬到过数据,现在work运行起来后 就爬取不到数据
2.期望爬取到数据
3.erron 和 warning
4. 最近一版,linux 操作系统, 有读

如下是启动work后出现的错误信息
[2020-08-23 13:54:09,820: ERROR/ForkPoolWorker-1] failed to crawl http://weibo.com/u/5818921523?is_ori=1&is_tag=0&profile_ftype=1&page=1,here are details:(535, b'5.7.12 SMTP access disabled'), stack is File "/home/gaoluoluo/Projects/weibo_work/decorators/decorator.py", line 14, in time_limit
return func(*args, **kargs)

2020-08-23 13:54:09 - crawler - WARNING - user 5818921523 has no weibo
[2020-08-23 13:54:09,822: WARNING/ForkPoolWorker-1] user 5818921523 has no weibo
[2020-08-23 13:54:09,826: INFO/MainProcess] Received task: tasks.home.crawl_weibo_datas[fa9e0c4a-1ecd-421a-99a1-2022353c202c]
2020-08-23 13:54:10 - crawler - ERROR - failed to crawl http://weibo.com/u/1696879551?is_ori=1&is_tag=0&profile_ftype=1&page=1,here are details:(535, b'5.7.12 SMTP access disabled'), stack is File "/home/gaoluoluo/Projects/weibo_work/decorators/decorator.py", line 14, in time_limit
return func(*args, **kargs)

[2020-08-23 13:54:10,126: ERROR/ForkPoolWorker-1] failed to crawl http://weibo.com/u/1696879551?is_ori=1&is_tag=0&profile_ftype=1&page=1,here are details:(535, b'5.7.12 SMTP access disabled'), stack is File "/home/gaoluoluo/Projects/weibo_work/decorators/decorator.py", line 14, in time_limit
return func(*args, **kargs)

2020-08-23 13:54:10 - crawler - WARNING - user 1696879551 has no weibo
[2020-08-23 13:54:10,128: WARNING/ForkPoolWorker-1] user 1696879551 has no weibo
[2020-08-23 13:54:10,132: INFO/MainProcess] Received task: tasks.home.crawl_weibo_datas[ba4fd564-c792-4a19-988e-5034129d25be]
[2020-08-23 13:54:10,186: INFO/MainProcess] Received task: tasks.home.crawl_weibo_datas[33470423-6e61-486d-b4b2-11bfc50aa708]
2020-08-23 13:57:30 - crawler - WARNING - user 6869376304 has no weibo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant