Implement Seed-level video capture setting handling + Job-level PDF-only option

[gretchenleighmiller]

This PR covers the following:

• Adds a new video_capture configuration option on the Seed level. This has four possible values; see newly added documentation for details.
• Implements the video_capture option, which impacts yt-dlp extraction and MIME types of outlinks. The remainder of video capture handling is accomplished in warcprox via Warcprox-Meta headers.
• Removes the previous file-based skip_av_seeds functionality.
• Adds a new pdfs_only configuration option on the Job level. This is a boolean that defaults to False; see newly added documentation for details.
• Implements the pdfs_only option based on the MIME type of outlinks. The remaining of PDF-only filtering is accomplished in warcprox via Warcprox-Meta headers.
• Documentation updates and minor style cleanup.

[galgeek]

with if and elif, shouldn't pdfs_only should get checked first?

[gretchenleighmiller]

It should have the correct behavior either way.

The first conditional is checking that 1) there is a rule to limit video capture by MIME type AND 2) that the MIME type specified in the Content-Type header of the response indicates that it's a video. Both have to be true to enter that branch, so we aren't going to accidentally skip the PDF conditional if Content-Type is a PDF.

The second conditional is similarly constrained on the PDF-only option being set and the Content-Type header specifying that the response is a PDF.

I would not expect the order to matter, but maybe I'm missing something?

[galgeek]

thanks! I did miss self._is_video_type(page_headers) initially.

I still like starting with the pdfs_only check — it seems simpler, and encompasses any limiting VideoCaptureOptions enabled.

[gretchenleighmiller]

Swapped them.

[galgeek]

Current code changes only processing of page-level urls with PDF-ish content-type.

[gretchenleighmiller]

Are you suggesting a better way of wording this in the documentation? It is called out here that this option specifically has limited impact and must be paired with a warcprox_meta rule to further filter by MIME type.

Do you think "PDF-ish" is better phrasing given the ambiguity of MIME type in Content-Type headers? I would expect MIME type to be pretty straightforward in this case given that PDFs are a pretty well-defined and well-known format.

[galgeek]

Something like this would be better, I think:

"Limits captures to PDFs, based on page's content-type header.
This value, for now, affects only brozzler's capture of page-level urls.

Note: fully limiting a crawl to PDFs only will also require updates to brozzler's Warcprox-Meta header and warcprox."

[gretchenleighmiller]

Reworded slightly for clarity.

[galgeek]

@gretchenleighmiller, thanks for your work on this!

I've left a couple of comments it might be good to address.