Skip to content

Commit

Permalink
[docs, setup.py] cleaning up after bsddb3 removal
Browse files Browse the repository at this point in the history
  • Loading branch information
pawelmhm committed Sep 17, 2021
1 parent c9dd29e commit 0c647a8
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 12 deletions.
6 changes: 6 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
Changes
=======

2.0.0 (2021-09-20)
------------------
* drop Python 2 support
* replace bsddb3 with Python's dbm for storing request data
* [dev workflow] changed from Travis to Github Actions

1.2.1 (2017-02-09)
------------------

Expand Down
13 changes: 4 additions & 9 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,16 @@ scrapy-deltafetch
:target: https://codecov.io/gh/scrapy-plugins/scrapy-deltafetch

This is a Scrapy spider middleware to ignore requests
to pages containing items seen in previous crawls of the same spider,
thus producing a "delta crawl" containing only new items.
to pages seen in previous crawls of the same spider,
thus producing a "delta crawl" containing only new requests.

This also speeds up the crawl, by reducing the number of requests that need
to be crawled, and processed (typically, item requests are the most CPU
intensive).

Requirements
============

DeltaFetch middleware depends on Python's bsddb3_ package.

On Ubuntu/Debian, you may need to install ``libdb-dev`` if it's not installed already.
DeltaFetch middleware uses Python's dbm_ package to store requests fingerprints.

.. _bsddb3: https://pypi.python.org/pypi/bsddb3
.. _dbm: https://docs.python.org/3/library/dbm.html


Installation
Expand Down
8 changes: 5 additions & 3 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,12 @@
'License :: OSI Approved :: BSD License',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
],
install_requires=['Scrapy>=1.1.0', 'bsddb3']
install_requires=['Scrapy>=1.1.0']
)

0 comments on commit 0c647a8

Please sign in to comment.