| Version: | 0.3 |
|---|
pylinkvalidator is a standalone and pure python link validator and crawler that traverses a web site and reports errors (e.g., 500 and 404 errors) encountered. The crawler can also download resources such as images, scripts and stylesheets.
pylinkvalidator's performance can be improved by installing additional libraries that require a C compiler, but these libraries are optional.
We created pylinkvalidator so that it could be executed in environments without access to a compiler (e.g., Microsoft Windows, some posix production environments) or with an old version of python (e.g., Centos).
pylinkvalidator is highly modular and has many configuration options, but the only required parameter is the starting url: pylinkvalidate.py http://www.example.com/
pylinkvalidator can also be used programmatically by calling one of the functions
in pylinkvalidator.api
Install pylinkvalidator with pip or easy_install:
pip install pylinkvalidator
Crawl all pages from a site and show progress:
pylinkvalidate.py -P http://www.example.com/
pylinkvalidator does not require external libraries if executed with python 2.x. It requires beautifulsoup4 if executed with python 3.x. It has been tested on python 2.6, python 2.7, and python 3.3.
For production use, it is strongly recommended to use lxml or html5lib because the default HTML parser provided by python is not very lenient.
These libraries can be installed to enable certain modes in pylinkvalidator:
- lxml
- beautifulsoup can use lxml to speed up the parsing of HTML pages. Because lxml requires C libraries, this is only an optional requirement.
- html5lib
- beautifulsoup can use html5lib to process incorrect or strange markup. It is slower than lxml, but believed to be more lenient.
- gevent
- this non-blocking io library enables pylinkvalidator to use green threads instead of processes or threads. gevent could potentially speed up the crawling speed on web sites with many small pages.
- cchardet
- this library speeds up the detection of document encoding.
This is a list of all available options. See the end of the README file for usage examples.
Usage: pylinkvalidate.py [options] URL ...
Options:
--version Show program's version number and exit
-h, --help Show this help message and exit
-V VERBOSE, --verbose=VERBOSE
Display debugging info
None: --verbose=0 (default)
Quiet: --verbose=1
Info: --verbose=2
Crawler Options:
These options modify the way the crawler traverses the site.
-O, --test-outside Fetch resources from other domains without crawling
them
-H ACCEPTED_HOSTS, --accepted-hosts=ACCEPTED_HOSTS
Comma-separated list of additional hosts to crawl
(e.g., example.com,subdomain.another.com)
-i IGNORED_PREFIXES, --ignore=IGNORED_PREFIXES
Comma-separated list of host/path prefixes to ignore
(e.g., www.example.com/ignore_this_and_after/)
-b, --ignore-bad-tel-urls
ignore badly formed tel URLs missing the leading +
sign, e.g., tel:1234567890 - only necessary for Python
> 2.6
-u USERNAME, --username=USERNAME
Username to use with basic HTTP authentication
-p PASSWORD, --password=PASSWORD
Password to use with basic HTTP authentication
-M, --multi each argument is considered to be a different site
-D HEADER, --header=HEADER
custom header of the form Header: Value (repeat for
multiple headers)
--url-file-path=URL_FILE_PATH
get starting URLs from a line-separated file
-t TYPES, --types=TYPES
Comma-separated values of tags to look for when
crawling a site. Default (and supported types):
a,img,link,script
-T TIMEOUT, --timeout=TIMEOUT
Seconds to wait before considering that a page timed
out (default = 10)
-C, --strict Does not strip href and src attributes from
whitespaces
-P, --progress Prints crawler progress in the console
-N, --run-once Only crawl the first page (eq. to depth=0)
-d DEPTH, --depth=DEPTH
Maximum crawl depth (default = 1)
-e, --prefer-server-encoding
Prefer server encoding if specified. Else detect
encoding
--check-presence=CONTENT_PRESENCE
Check presence of raw or HTML content on all pages.
e.g., <tag attr1="val">regex:content</tag>. Content
can be either regex:pattern or plain content
--check-absence=CONTENT_ABSENCE
Check absence of raw or HTML content on all pages.
e.g., <tag attr1="val">regex:content</tag>. Content
can be either regex:pattern or plain content
--check-presence-once=CONTENT_PRESENCE_ONCE
Check presence of raw or HTML content for one page:
path,content, e.g.,: /path,<tag
attr1="val">regex:content</tag>. Content can be either
regex:pattern or plain content. Path can be either
relative or absolute with domain.
--check-absence-once=CONTENT_ABSENCE_ONCE
Check absence of raw or HTML content for one page:
path,content, e.g.,path,<tag
attr1="val">regex:content</tag>. Content can be either
regex:pattern or plain content. Path can be either
relative or absolute with domain.
-S, --show-source Show source of links (html) in the report.
Performance Options:
These options can impact the performance of the crawler.
-w WORKERS, --workers=WORKERS
Number of workers to spawn (default = 1)
-m MODE, --mode=MODE
Types of workers: thread (default), process, or green
-R PARSER, --parser=PARSER
Types of HTML parse: html.parser (default) or lxml
Output Options:
These options change the output of the crawler.
-f FORMAT, --format=FORMAT
Format of the report: plain (default)
-o OUTPUT, --output=OUTPUT
Path of the file where the report will be printed.
-W WHEN, --when=WHEN
When to print the report. error (only if a
crawling error occurs) or always (default)
-E REPORT_TYPE, --report-type=REPORT_TYPE
Type of report to print: errors (default, summary and
erroneous links), summary, all (summary and all links)
-c, --console Prints report to the console in addition to other
output options such as file or email.
Email Options:
These options allows the crawler to send a report by email.
-a ADDRESS, --address=ADDRESS
Comma-separated list of email addresses used to send a
report
--from=FROM_ADDRESS
Email address to use in the from field of the email
(optional)
-s SMTP, --smtp=SMTP
Host of the smtp server
--port=PORT Port of the smtp server (optional)
--tls Use TLS with the email server.
--subject=SUBJECT Subject of the email (optional)
--smtp-username=SMTP_USERNAME
Username to use with the smtp server (optional)
--smtp-password=SMTP_PASSWORD
Password to use with the smtp server (optional)
- Crawl a site and show progress
pylinkvalidate.py --progress http://example.com/- Crawl a site starting from 2 URLs
pylinkvalidate.py http://example.com/ http://example2.com/- Crawl a site (example.com) and all pages belonging to another host
pylinkvalidate.py -H additionalhost.com http://example.com/- Report status of all links (even successful ones)
pylinkvalidate.py --report-type=all http://example.com/- Report status of all links and HTML show source of these links
pylinkvalidate.py --report-type=all --show-source http://example.com/- Only crawl starting URLs and access all linked resources
pylinkvalidate.py --run-once http://example.com/- Crawl two levels (one more than run-once) and access all linked resources
pylinkvalidate.py --depth=1 http://example.com/- Only access links (a href) and ignore images, stylesheets and scripts
pylinkvalidate.py --types=a http://example.com/- Crawl a site with 4 threads (default is one thread)
pylinkvalidate.py --workers=4 http://example.com/- Crawl a site with 4 processes (default is one thread)
pylinkvalidate.py --mode=process --workers=4 http://example.com/- Crawl a site and use LXML to parse HTML (faster, must be installed)
pylinkvalidate.py --parser=LXML http://example.com/- Print debugging info
pylinkvalidate.py --verbose=2 http://example.com/- Change User-Agent request header
pylinkvalidate.py --header="User-Agent: Mozilla/5.0" http://example.com/- Crawl multiple sites and report results per site
pylinkvalidate.py --multi http://example.com/ http://www.example2.net/- Check that all HTML pages have a body tag with a specific class:
pylinkvalidate.py --check-content '<body class="test"></body>' http://example.com/- Check that no HTML pages have a paragraph tag with a pattern:
pylinkvalidate.py --check-absence '<p>regex:Hello\s+World</body>' http://example.com/- Check that robots.txt have a Disallow none:
pylinkvalidate.py --check-content-once '/robots.txt,regex:^Disallow:\s*$' http://example.com/
To crawl a site from a single URL:
from pylinkvalidator.api import crawl
crawled_site = crawl("http://www.example.com/")
number_of_crawled_pages = len(crawled_site.pages)
number_of_errors = len(crawled_sites.error_pages)To crawl a site and pass some configuration options (the same supported by the command line interface):
from pylinkvalidator.api import crawl_with_options
crawled_site = crawl_with_options(["http://www.example.com/"], {"run-once":
True, "workers": 10})
number_of_crawled_pages = len(crawled_site.pages)
number_of_errors = len(crawled_sites.error_pages)- I cannot find pylinkvalidate.py on Windows with virtualenv
- This is a known problem with virtualenv on windows. The interpreter is
different than the one used by the virtualenv. Prefix pylinkvalidate.py with the
full path:
python c:\myvirtualenv\Scripts\pylinkvalidate.py - I see Exception KeyError ... module 'threading' when using --mode=green
- This output is generally harmless and is generated by gevent patching the python thread module. If someone knows how to make it go away, patches are more than welcome :-)
This software is licensed under the New BSD License. See the LICENSE file in the for the full license text. It includes the beautifulsoup library which is licensed under the MIT license.