Skip to content

Commit

Permalink
[update] extend_fetcher
Browse files Browse the repository at this point in the history
  • Loading branch information
jhao104 committed May 27, 2021
1 parent 409dae4 commit 202b427
Show file tree
Hide file tree
Showing 4 changed files with 46 additions and 3 deletions.
4 changes: 2 additions & 2 deletions docs/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ ChangeLog
------------------

1. 修复Dockerfile时区问题; (2021-04-12)
2. 新增Proxy属性 `source`, 标记代理来源; (2021-04-13)
3. 新增Proxy属性 `https`, 标记支持https的代理; (2021-05-27)
2. 新增Proxy属性 ``source``, 标记代理来源; (2021-04-13)
3. 新增Proxy属性 ``https``, 标记支持https的代理; (2021-05-27)

2.2.0 (2021-04-08)
------------------
Expand Down
34 changes: 33 additions & 1 deletion docs/dev/extend_fetcher.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,36 @@
.. extend_fetcher
扩展代理源
-----------
-----------

项目默认包含几个免费的代理获取源,但是免费的毕竟质量有限,如果直接运行可能拿到的代理质量不理想。因此提供了用户自定义扩展代理获取的方法。

如果要添加一个新的代理获取方法, 过程如下:

1. 首先在 `ProxyFetcher`_ 类中添加自定义的获取代理的静态方法,该方法需要以生成器(yield)形式返回 ``host:ip`` 格式的代理字符串, 例如:

.. code-block:: python
class ProxyFetcher(object):
# ....
# 自定义代理源获取方法
@staticmethod
def freeProxyCustom01(): # 命名不和已有重复即可
# 通过某网站或者某接口或某数据库获取代理
# 假设你已经拿到了一个代理列表
proxies = ["x.x.x.x:3128", "x.x.x.x:80"]
for proxy in proxies:
yield proxy
# 确保每个proxy都是 host:ip正确的格式返回
2. 添加好方法后,修改配置文件 `setting.py`_ 中的 ``PROXY_FETCHER`` 项, 加入刚才添加的自定义方法的名字:

.. code-block:: python
PROXY_FETCHER = [
# ....
"freeProxyCustom01" # # 确保名字和你添加方法名字一致
]
.. _ProxyFetcher: https://github.com/jhao104/proxy_pool/blob/1a3666283806a22ef287fba1a8efab7b94e94bac/fetcher/proxyFetcher.py#L20
.. _setting.py: https://github.com/jhao104/proxy_pool/blob/1a3666283806a22ef287fba1a8efab7b94e94bac/setting.py#L47
10 changes: 10 additions & 0 deletions docs/dev/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
=========
开发指南
=========

.. module:: dev

.. toctree::
:maxdepth: 2

extend_fetcher
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -112,4 +112,5 @@ Contents
:maxdepth: 2

user/index
dev/index
changelog

0 comments on commit 202b427

Please sign in to comment.