Skip to content

cdegroc/warc-clueweb

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

99 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

warc-clueweb: Python library to work with ClueWeb09 WARC files

ClueWeb09 dataset consists of ~1 billion web pages crawled in January and February 2009. It is used by several tracks of the TREC conference.

The dataset is available in the WARC 0.18 format. However, there are certain caveats with ClueWeb's files. First, it does not use the standard \r\n end-of-line markers. Moreover, some records are ill-formed.

This library is a fork of internetarchive/warc that can handle ClueWeb09 dataset.

Documentation

Only minor modifications to the original library were made. The original documentation of the warc library still holds. It is available at http://warc.readthedocs.org/.

License

This software is licensed under GPL v2. See LICENSE file for details.

About

Python library for reading ClueWeb09's warc files

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%