ClueWeb09 dataset consists of ~1 billion web pages crawled in January and February 2009. It is used by several tracks of the TREC conference.
The dataset is available in the WARC 0.18 format. However, there are certain caveats with ClueWeb's files. First, it does not use the standard \r\n end-of-line markers. Moreover, some records are ill-formed.
This library is a fork of internetarchive/warc that can handle ClueWeb09 dataset.
Only minor modifications to the original library were made. The original documentation of the warc library still holds. It is available at http://warc.readthedocs.org/.
This software is licensed under GPL v2. See LICENSE file for details.