This contains rules for https://github.com/zytedata/duplicate-url-discarder.
pip install duplicate-url-discarder-rules
The rules can be imported via:
from duplicate_url_discarder_rules import RULE_PATHS
It can then be used to configure the DUD_LOAD_RULE_PATHS
setting of
duplicate-url-discarder
.