[NUTCH-2865] WARC exporter support for metadata and dropping empty responses

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Fix Version/s: 1.19
• Patch Info: Patch Available
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 20/May/21 10:05
• Updated: 13/Mar/24 14:50
• Resolved: 22/Nov/21 13:53

Description

WARCExporter is a handy tool to dump the segments. Unfortunately it also emits WARC records for status' other than success of notmodified, which accounts for a decent number in each crawl cycle. It also doesn't emit parsed metadata or extracted text. It does now.

 

This patch adds three switches:

• -includeOnlySuccessfulResponses to only emit records of success or notmodified
• -includeParseData to also emit parse metadata as WARC metadata record
• -includeParseText to also emit extracted text as WARC metadata

Both metadata objects are stored in the same WARC metadata record to save space.

Issue Links

• links to: GitHub Pull Request #706